An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2338–2344 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2338 An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model Kijong Han1∗, Seojin Lee2∗†, Wooin Lee1, Joosung Lee1, and Dong-hun Lee1‡ 1Kakao Enterprise, South Korea 2SK Telecom, South Korea {mat.h,dan.kes,rung.joo,hubert.std}@kakaoenterprise.com seojin.lee@sktair.com Abstract Multi-turn response selection models have re- cently shown comparable performance to hu- mans in several benchmark datasets. However, in the real environment, these models often have weaknesses, such as making incorrect predictions based heavily on superficial pat- terns without a comprehensive understanding of the context. For example, these models of- ten give a high score to the wrong response candidate containing several keywords related to the context but using the inconsistent tense. In this study, we analyze the weaknesses of the open-domain Korean Multi-turn response selection models and publish an adversarial dataset to evaluate these weaknesses. We also suggest a strategy to build a robust model in this adversarial environment. 1 Introduction Multi-turn response selection is a task that selects the best response among given candidates for a given dialogue context. Response selection mod- els have recently shown comparable performance to humans (Cui et al., 2020) in the several in- domain/held-out benchmarks (Lowe et al., 2015; Zhang et al., 2018a; Dinan et al., 2020). However, in the actual service environment, these models are often found to have weaknesses. For example, the model gives the highest score to the wrong response, which has high word overlap with the context (Yuan et al., 2019) or semantically similar to the context (Whang et al., 2021). Held-out evaluation often overestimates the real- world performance of the model (Ribeiro et al., 2020), so adversarial datasets for evaluating weak- nesses have been constructed for each task, such as NLI (Naik et al., 2018; McCoy et al., 2019), and MRC (Jia and Liang, 2017; Rajpurkar et al., 2018). ∗These authors contributed equally to this work. † This work was done while the author was working at Kakao Enterprise. ‡Corresponding author. A framework for comprehensively evaluating the general linguistic abilities of the model was also studied (Ribeiro et al., 2020). Several works evaluated adversarial cases for the response selection task (Yuan et al., 2019; Whang et al., 2021). However, they just automatically gen- erate adversarial responses by copying words in the context. In this study, we analyze the weak- nesses of the various aspects of the open-domain Korean Multi-turn Response Selection models and construct an adversarial dataset manually. A total of 2,220 test cases are constructed, and each test case are classified by type. Neural networks do not generalize well to such an adversarial setting because they tend to use su- perficial patterns and spurious correlation of the dataset overly, which makes models biased(Clark et al., 2019; Nam et al., 2020). Thus, various debi- asing methods have been studied to alleviate this phenomenon (He et al., 2019; Utama et al., 2020). In this study, we show that debasing method is also …

Content truncated to 500 words