DuQM: A Chinese Dataset of Linguistically Perturbed Natural Questions for Evaluating the Robustness of Question Matching Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7782 - 7794 December 7-11, 2022 ©2022 Association for Computational Linguistics DuQM: A Chinese Dataset of Linguistically Perturbed Natural Questions for Evaluating the Robustness of Question Matching Models Hongyu Zhu♠♣†, Yan Chen♣†, Jing Yan♣†, Jing Liu♣∗, Yu Hong♠∗, Ying Chen♣, Hua Wu♣, Haifeng Wang♣ ♠ School of Computer Science and Technology, Soochow University, China ♣ Baidu Inc., Beijing, China {hines.zhu, tianxianer}@gmail.com {chenyan22, yanjing09, liujing46, chenying04, wu_hua, wanghaifeng}@baidu.com Abstract In this paper, we focus on the robustness eval- uation of Chinese Question Matching (QM) models. Most of the previous work on ana- lyzing robustness issues focus on just one or a few types of artificial adversarial examples. Instead, we argue that a comprehensive eval- uation should be conducted on natural texts, which takes into account the fine-grained lin- guistic capabilities of QM models. For this purpose, we create a Chinese dataset namely DuQM which contains natural questions with linguistic perturbations to evaluate the robust- ness of QM models. DuQM contains 3 cate- gories and 13 subcategories with 32 linguis- tic perturbations. The extensive experiments demonstrate that DuQM has a better ability to distinguish different models. Importantly, the detailed breakdown of evaluation by the lin- guistic phenomenon in DuQM helps us easily diagnose the strength and weakness of different models. Additionally, our experiment results show that the effect of artificial adversarial ex- amples does not work on natural texts. Our baseline codes and a leaderboard are now pub- licly available.1 1 Introduction The task of Question Matching (QM) aims to iden- tify the question pairs that have the same meaning, and it has been widely used in many applications, e.g., community question answering and intelli- gent customer services, etc. Though neural QM models have shown compelling performance on various datasets, including Quora Question Pairs (QQP) (Iyer et al., 2017), LCQMC (Liu et al., 2018), BQ (Chen et al., 2018) and AFQMC2, neu- †Equal contribution. The work was done when Hongyu Zhu was doing internship at Baidu. ∗Co-corresponding authors. 1Code: https://github.com/baidu/DuReader/tree/master/Du QM; leaderboard: https://aistudio.baidu.com/aistudio/compe tition/detail/116/0/introduction. 2It is from Ant Technology Exploration Conference (ATEC) Developer competition, which is not available now. ral models are often not robust to adversarial exam- ples, which means that the neural models predict unexpected outputs given just a small perturbations on the inputs. As the example 1 in Tab. 1 shows, a model might not distinguish the minor difference ("面 noodles") between the two sentences, and thus predicts the two questions semantically equivalent. Recently, it attracts a lot of attentions from the research community to deal with the robustness issues of neural models on various NLP tasks, such as question matching, natural language inference and machine reading comprehension. Early works examine the robustness of neural models by cre- ating certain types of artificial adversarial exam- ples (Jia and Liang, 2017; Alzantot et al., 2018; Ren et al., 2019; Jin et al., 2020), and involving human-and-model-in-the-loop to create dynamic adversarial examples (Nie et al., 2020; Wallace et …

Content truncated to 500 words