Robustness Testing of Language Understanding in Task-Oriented Dialog

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2467–2480 August 1–6, 2021. ©2021 Association for Computational Linguistics 2467 Robustness Testing of Language Understanding in Task-Oriented Dialog Jiexi Liu1∗, Ryuichi Takanobu1∗, Jiaxin Wen1, Dazhen Wan1, Hongguang Li2, Weiran Nie2, Cheng Li2, Wei Peng2, Minlie Huang1† 1CoAI Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China 2Huawei Technologies, Shenzhen, China {liujiexi19, gxly19}@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Most language understanding models in task- oriented dialog systems are trained on a small amount of annotated training data, and evalu- ated in a small set from the same distribution. However, these models can lead to system fail- ure or undesirable output when being exposed to natural language perturbation or variation in practice. In this paper, we conduct compre- hensive evaluation and analysis with respect to the robustness of natural language understand- ing models, and introduce three important as- pects related to language understanding in real- world dialog systems, namely, language vari- ety, speech characteristics, and noise pertur- bation. We propose a model-agnostic toolkit LAUG to approximate natural language pertur- bations for testing the robustness issues in task- oriented dialog. Four data augmentation ap- proaches covering the three aspects are assem- bled in LAUG, which reveals critical robust- ness issues in state-of-the-art models. The aug- mented dataset through LAUG can be used to facilitate future research on the robustness test- ing of language understanding in task-oriented dialog. 1 Introduction Recently task-oriented dialog systems have been at- tracting more and more research efforts (Gao et al., 2019; Zhang et al., 2020b), where understanding user utterances is a critical precursor to the suc- cess of such dialog systems. While modern neural networks have achieved state-of-the-art results on language understanding (LU) (Wang et al., 2018; Zhao and Feng, 2018; Goo et al., 2018; Liu et al., 2019; Shah et al., 2019), their robustness to changes in the input distribution is still one of the biggest challenges in practical use. ∗Equal contribution. †Corresponding author. Real dialogs between human participants in- volve language phenomena that do not contribute so much to the intent of communication. As shown in Fig. 1, user expressions can be of high lexical and syntactic diversity when a system is deployed to users; typed texts may differ significantly from those recognized from voice speech; interaction environments may be full of chaos and even users themselves may introduce irrelevant noises such that the system can hardly get clean user input. Unfortunately, neural LU models are vulnerable to these natural perturbations that are legitimate inputs but not observed in training data. For ex- ample, Bickmore et al. (2018) found that popular conversational assistants frequently failed to under- stand real health-related scenarios and were unable to deliver adequate responses on time. Although many studies have discussed the LU robustness (Ray et al., 2018; Zhu et al., 2018; Iyyer et al., 2018; Yoo et al., 2019; Ren et al., 2019; Jin et al., 2020; He et al., 2020), there …

Content truncated to 500 words