Trends and methods in chatbot evaluation

Casas, Jacky (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Tricot, Marc-Olivier (University of Fribourg, Fribourg, Switzerland) ; Khaled, Omar Abou (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Mugellini, Elena (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Cudré-Mauroux, Philippe (University of Fribourg, Fribourg, Switzerland)

Chatbots are computer programs aiming to replicate human conversational abilities through voice exchanges, textual dialogues, or both. They are becoming increasingly pervasive in many domains like customer support, e-coaching or entertainment. Yet, there is no standardised way of measuring the quality of such virtual agents. Instead, multiple individuals and groups have established their own standards either specifically for their chatbot project or have taken some inspiration from other groups. In this paper, we make a review of current techniques and trends in chatbot evaluation. We examine chatbot evaluation methodologies and assess them according to the ISO 9214 concepts of usability: Effectiveness, Efficiency and Satisfaction. We then analyse the methods used in the literature from 2016 to 2020 and compare their results. We identify a clear trend towards evaluating the efficiency of chatbots in many recent papers, which we link to the growing popularity of task-based chatbots that are currently being deployed in many business contexts.


Mots-clés:
Type de conférence:
non-published full paper
Faculté:
Ingénierie et Architecture
Ecole:
HEIA-FR
Institut:
HumanTech - Technology for Human Wellbeing Institute
Adresse bibliogr.:
Utrecht, The Netherlands, 25-29 October 2020
Date:
2020-10
Utrecht, The Netherlands
25-29 October 2020
Pagination:
7 p.
Publié dans:
Proceedings of ICMI 2020 : 22nd ACM International Conference on Multimodal Interaction, 25-29 October 2020, Utrecht, the Netherlands
DOI:
ISBN:
9781450380027
Le document apparaît dans:

Note: The status of this file is: restricted


 Notice créée le 2020-12-15, modifiée le 2020-12-16

Fulltext:
Télécharger le document
PDF

Évaluer ce document:

Rate this document:
1
2
3
 
(Pas encore évalué)