Interactive Evaluation of Dialog Track at DSTC9

5731 Interactive Evaluation of Dialog Track at DSTC9 Shikib Mehri1, Yulan Feng1, Carla Gordon2, Seyed Hossein Alavi2, David Traum2, Maxine Eskenazi1 1Language Technologies Institute, Carnegie Mellon University 2Institute for Creative Technologies, University of Southern California amehri@andrew.cmu.edu Abstract The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategies that ex- tend them to back-and-forth interactions with real users. The progression from static corpora to interactive evaluation in- troduces unique challenges and facilitates a more thorough assessment of open-domain dialog systems. This paper pro- vides an overview of the track, including the methodology and results. Furthermore, it provides insights into how to best evaluate open-domain dialog models. Introduction A long-standing challenge in computer science is the develop- ment of algorithms that can interact with humans in natural language (Turing 1950). Ultimately, the goal of dialog re- search is to create systems that can engage in back-and-forth interactions with real users (Eskenazi et al. 2019). However, the majority of research is performed on static datasets. For example, the task of response generation is typically done by producing a response for a static dialog context (Vinyals and Le 2015). By reducing dialog to response generation, static evaluation neglects multiple important challenges of dialog. In contrast, interactive evaluation allows several valuable properties of dialog to be measured, including: consistency, topic depth, adaptation, error recovery and user-centric devel- opment. Mehri and Eskenazi (2020a) found that state-of-the- art dialog models perform on-par with humans on response generation, but they fall short when considering an entire dialog. To promote interactive evaluation of dialog, the Inter- active Evaluation of Dialog Track of the 9th Dialog System Technology Challenge (Gunasekara et al. 2020) challenged participants to build models for open-domain interaction with real users. Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. This track consists of two sub-tasks: (1) static evaluation and (2) interactive evaluation. The goal of the first sub-task is to develop knowledge-grounded response generation models which are then evaluated in a static manner using the Topical- Chat corpus (Gopalakrishnan et al. 2019). The second sub- task challenges participants to extend response generation models to effectively converse with real users through the Di- alPort portal (Zhao, Lee, and Eskenazi 2016). Through these two sub-tasks, the track challenges participants to first de- velop strong response generation models and then to explore strategies for extending them to interactive settings. In the following sections, we describe the methodology and results for both sub-tasks. We then present insights into methods of best evaluating open-domain dialog models. Related Work …

Content truncated to 500 words