Deep multimodal case–based retrieval for large histopathology datasets

Jimenez-del-Toro, Oscar (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) ; University of Geneva (UNIGE) Switzerland) ; Otálora, Sebastian (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) ; University of Geneva (UNIGE) Switzerland) ; Atzori, Manfredo (University of Geneva (UNIGE) Switzerland) ; Müller, Henning (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis) ; University of Geneva (UNIGE) Switzerland)

The current gold standard for interpreting patient tissue samples is the visual inspection of whole–slide histopathology images (WSIs) by pathologists. They generate a pathology report describing the main findings relevant for diagnosis and treatment planning. Search-ing for similar cases through repositories for differential diagnosis is often not done due to a lack of efficient strategies for medical case–based re-trieval. A patch–based multimodal retrieval strategy that retrieves sim-ilar pathology cases from a large data set fusing both visual and text information is explained in this paper. By fine–tuning a deep convolu-tional neural network an automatic representation is obtained for the vi-sual content of weakly annotated WSIs (using only a global cancer score and no manual annotations). The pathology text report is embedded into a category vector of the pathology terms also in a non–supervised approach. A publicly available data set of 267 prostate adenocarcinoma cases with their WSIs and corresponding pathology reports was used to train and evaluate each modality of the retrieval method. A MAP (Mean Average Precision) of 0.54 was obtained with the multimodal method in a previously unseen test set. The proposed retrieval system can help in differential diagnosis of tissue samples and during the training of pathol-ogists, exploiting the large amount of pathology data already existing digital hospital repositories.


Faculty:
Economie et Services
School:
HEG-VS
Institute:
Institut Informatique de gestion
Subject(s):
Informatique
Publisher:
Cham, Springer
Date:
Cham
Springer
2017
Pagination:
pp. 149-157
Published in:
Patch-based techniques in medical imaging : third International Workshop, Patch-MI 2017, held in conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, Proceedings
DOI:
ISBN:
978-3-319-67433-9
External resources:
Appears in Collection:

Note: The status of this file is: restricted


 Record created 2017-11-11, last modified 2019-11-28

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)