VISCERAL : evaluation-as-a-service for medical imaging

Hanbury, Allan (TU WienInstitute of Software Technology and Interactive Systems, Vienna, Austria) ; Müller, Henning (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis))

Systematic evaluation has had a strong impact on many data analysis domains, for example, TREC and CLEF in information retrieval, ImageCLEF in image retrieval, and many challenges in conferences such as MICCAI for medical imaging and ICPR for pattern recognition. With Kaggle, a platform for machine learning challenges has also had a significant success in crowdsourcing solutions. This shows the importance to systematically evaluate algorithms and that the impact is far larger than simply evaluating a single system. Many of these challenges also showed the limits of the commonly used paradigm to prepare a data collection and tasks, distribute these and then evaluate the participants’ submissions. Extremely large datasets are cumbersome to download, while shipping hard disks containing the data becomes impractical. Confidential data can often not be shared, for example medical data, and also data from company repositories. Real-time data will never be available via static data collections as the data change over time and data preparation often takes much time. The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of these problems and has been applied in the VISCERAL project. In EaaS, the data are not moved but remain on a central infrastructure. In the case of VISCERAL, all data were made available in a cloud environment. Participants were provided with virtual machines on which to install their algorithms. Only a small part of the data, the training data, was visible to participants. The major part of the data, the test data, was only accessible to the organizers who ran the algorithms in the participants’ virtual machines on the test data to obtain impartial performance measures.

Economie et Services
HEG VS HES-SO Valais-Wallis - Haute Ecole de Gestion & Tourisme
Institut Informatique de gestion
Adresse bibliogr.:
Cham, Springer
pp. 3-13
Publié dans
Cloud-based benchmarking of medical image analysis
Ressource(s) externe(s):
Le document apparaît dans:

Note: The status of this file is: restricted

 Notice créée le 2017-10-23, modifiée le 2018-11-09

Télécharger le document

Évaluer ce document:

Rate this document:
(Pas encore évalué)