Deep multimodal classification of image types in biomedical journal figures

Adrearczyk, Vincent (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis)) ; Müller, Henning (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) ; University of Geneva, Switzerland)

This paper presents a robust method for the classification of medical image types in figures of the biomedical literature using the fusion of visual and textual information. A deep convolutional network is trained to discriminate among 31 classes including compound figures, diagnostic image types and generic illustrations, while another shallow convolutional network is used for the analysis of the captions paired with the images. Various fusion methods are analyzed as well as data augmentation approaches. The proposed system is validated on the ImageCLEF 2013 classification task, largely improving the currently best performance from 83.5% to 93.7% accuracy.


Faculty:
Economie et Services
School:
HEG-VS
Institute:
Institut Informatique de gestion
Subject(s):
Informatique
Publisher:
Cham, Springer
Date:
2018-09
Cham
Springer
Pagination:
Pp. 3-14
Published in:
Experimental IR meets multilinguality, multimodality, and interaction : 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, September 10-14, 2018, Proceedings
Author of the book:
Bellot, P. ; (ed.)
Trabelsi, C. ; (ed.)
Mothe, J ; (ed.)
DOI:
ISBN:
978-3-319-98931-0
External resources:
Appears in Collection:

Note: The status of this file is: restricted


 Record created 2018-11-16, last modified 2018-12-20

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)