RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4418–4429 August 1–6, 2021. ©2021 Association for Computational Linguistics 4418 RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems Baolin Peng1, Chunyuan Li1, Zhu Zhang12† , Chenguang Zhu1, Jinchao Li1, Jianfeng Gao1 1Microsoft Research, Redmond, WA 2Iowa State University / Ames, IA {bapeng,chunyl,chezhu,jincli,jfgao}@microsoft.com zhuzhang@iastate.edu Abstract For task-oriented dialog systems to be maxi- mally useful, it must be able to process con- versations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities, or domains. In pursuit of these goals, we introduce the RAD- DLE1 benchmark 2, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains. By including tasks with limited training data, RADDLE is designed to favor and encourage models with a strong generalization ability. RADDLE also includes a diagnostic checklist that facilitates detailed robustness analysis in aspects such as language variations, speech errors, unseen entities, and out-of-domain utterances. We evaluate recent state-of-the-art systems based on pre-training and fine-tuning, and find that grounded pre-training on heterogeneous dia- log corpora performs better than training a sep- arate model per domain. Adversarial training is also proposed to improve model robustness against noisy inputs. Overall, existing models are less than satisfactory in robustness evalu- ation, which suggests opportunities for future improvement. 1 Introduction Dialogs constitute a crucial communication chan- nel in completing a broad range of tasks, such as weather query, flight and restaurant booking, movie booking, IT help desk, etc. Comparing to chit- chat systems that are usually modeled with single- turn context-response pairs, task-oriented dialog systems involve retrieving information from knowl- edge bases and reasoning over multiple dialog turns. This makes it especially important for a system to †Work was done when Zhu Zhang was visiting MSR 1Robust tAsk-orienteD DiaLog systems Evaluation 2Benchmark link: http://aka.ms/raddle be able to produce response that are grounded on tasks goals and user intents. In a bid to support human-computer interactions, task-oriented dialog systems have been built to allow users to converse with a computer system using natural language, such as Siri, Google Assistant, Amazon Alexa, Mi- crosoft XiaoIce (Zhou et al., 2020). Traditionally, a task-oriented dialog system uses a modularized pipeline with four modules that execute sequen- tially (Gao et al., 2019). A natural language un- derstanding (NLU) module identifies user intents and extracts associated information such as slots and corresponding values from user input. A dia- log state tracker (DST) infers the belief state (or user goal) from dialog history. The belief state is often used to query a task-specific database (DB) to obtain the DB state, such as the number of enti- ties that match the user goal. The dialog state and DB state are then passed to a dialog policy (POL) module to select the next system action. A …

Content truncated to 500 words