Trends and methods in chatbot evaluation

Casas, Jacky (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Tricot, Marc-Olivier (University of Fribourg, Fribourg, Switzerland) ; Khaled, Omar Abou (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Mugellini, Elena (School of Engineering and Architecture (HEIA-FR), HES-SO // University of Applied Sciences Western Switzerland) ; Cudré-Mauroux, Philippe (University of Fribourg, Fribourg, Switzerland)

Chatbots are computer programs aiming to replicate human conversational abilities through voice exchanges, textual dialogues, or both. They are becoming increasingly pervasive in many domains like customer support, e-coaching or entertainment. Yet, there is no standardised way of measuring the quality of such virtual agents. Instead, multiple individuals and groups have established their own standards either specifically for their chatbot project or have taken some inspiration from other groups. In this paper, we make a review of current techniques and trends in chatbot evaluation. We examine chatbot evaluation methodologies and assess them according to the ISO 9214 concepts of usability: Effectiveness, Efficiency and Satisfaction. We then analyse the methods used in the literature from 2016 to 2020 and compare their results. We identify a clear trend towards evaluating the efficiency of chatbots in many recent papers, which we link to the growing popularity of task-based chatbots that are currently being deployed in many business contexts.


Keywords:
Conference Type:
non-published full paper
Faculty:
Ingénierie et Architecture
School:
HEIA-FR
Institute:
HumanTech - Technology for Human Wellbeing Institute
Publisher:
Utrecht, The Netherlands, 25-29 October 2020
Date:
2020-10
Utrecht, The Netherlands
25-29 October 2020
Pagination:
7 p.
Published in:
Proceedings of ICMI 2020 : 22nd ACM International Conference on Multimodal Interaction, 25-29 October 2020, Utrecht, the Netherlands
DOI:
ISBN:
9781450380027
Appears in Collection:

Note: The status of this file is: restricted


 Record created 2020-12-15, last modified 2020-12-16

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)