EaaS : evaluation-as-a-service and experiences from the VISCERAL project

Müller, Henning (University of Applied Sciences and Arts Western Switzerland (HES-SO Valais-Wallis)) ; Hanbury, Allan (TU Wien, Austria)

The Cranfield paradigm has dominated information retrieval evaluation for almost 50 years. It has had a major impact on the entire domain of information retrieval since the 1960s and, compared with systematic evaluation in other domains, is very well developed and has helped very much to advance the field. This chapter summarizes some of the shortcomings in information analysis evaluation and how recent techniques help to leverage these shortcomings. The term Evaluation-as-a-Service (EaaS) was defined at a workshop that combined several approaches that do not distribute the data but use source code submission, APIs or the cloud to run evaluation campaigns. The outcomes of a white paper and the experiences gained in the VISCERAL project on cloud-based evaluation for medical imaging are explained in this paper. In the conclusions, the next steps for research infrastructures are imagined and the impact that EaaS can have in this context to make research in data science more efficient and effective.


Faculty:
Economie et Services
School:
HEG-VS
Institute:
Institut Informatique de gestion
Subject(s):
Informatique
Publisher:
Cham, Springer
Date:
2019-08
Cham
Springer
Pagination:
pp. 161-173
Published in:
Information retrieval evaluation in a changing world : lessons learned from 20 Years of CLEF
Series Statement:
The Information Retrieval Series, vol. 41
Author of the book:
Ferro, Nicola ; Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova Italy
Peters, Carol ; Consiglio Nazionale delle Ricerche, Istituto di Scienza e Tecnologie dell’Informazione Pisa Italy
DOI:
ISSN:
1387-5264
ISBN:
978-3-030-22947-4
Appears in Collection:

Note: The status of this file is: restricted


 Record created 2019-10-25, last modified 2020-03-25

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)