Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasets

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4418–4429 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4418 Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasets Guanhua Zhang1,2∗, Bing Bai1∗, Jian Liang1, Kun Bai1, Shiyu Chang3, Mo Yu3, Conghui Zhu2, Tiejun Zhao2 1Cloud and Smart Industries Group, Tencent, China 2Harbin Institute of Technology, China 3MIT-IBM Watson AI Lab, IBM Research, USA {guanhzhang,icebai,joshualiang,kunbai}@tencent.com, shiyu.chang@ibm.com, yum@us.ibm.com, {chzhu,tjzhao}@hit-mtlab.net Abstract Natural Language Sentence Matching (NLSM) has gained substantial attention from both academics and the industry, and rich public datasets contribute a lot to this process. However, biased datasets can also hurt the generalization performance of trained models and give untrustworthy evaluation results. For many NLSM datasets, the providers select some pairs of sentences into the datasets, and this sampling procedure can easily bring unintended pattern, i.e., selection bias. One example is the QuoraQP dataset, where some content-independent na¨ıve features are unreasonably predictive. Such features are the reflection of the selection bias and termed as the “leakage features.” In this paper, we investigate the problem of selection bias on six NLSM datasets and find that four out of them are significantly biased. We further propose a training and evaluation framework to alleviate the bias. Experimental results on QuoraQP suggest that the proposed framework can improve the generalization ability of trained models, and give more trustworthy evaluation results for real-world adoptions. 1 Introduction Natural Language Sentence Matching (NLSM) aims at comparing two sentences and identifying the relationships (Wang et al., 2017), and serves as the core of many NLP tasks such as question answering and information retrieval (Wang et al., 2016b). Natural Language Inference (NLI) (Bow- man et al., 2015) and Semantic Textual Similar- ity (STS) (Wang et al., 2016b) are both typical NLSM problems. A large number of publicly available datasets have benefited the research to a great extent (Kim et al., 2018; Wang et al., * Equal contributions from both authors. This work was done when Guanhua Zhang was an intern at Tencent. negative WMD S1_freq S2_freq S1S2_inter 0.0 0.5 1.0 Normalized feature 0 1 Figure 1: Visualization for the distributions of nor- malized features versus the label in QuoraQP. The right part (in red) represents the distributions of duplicated pairs, and the left part (in blue) rep- resents the distributions of not duplicated pairs. Best viewed in color. 2017; Tien et al., 2018), including QuoraQP1, SNLI (Bowman et al., 2015), SICK (Marelli et al., 2014), etc. These datasets provide resources for both training and evaluation of different algo- rithms (Torralba and Efros, 2011). However, most of the datasets are prepared by conducting procedures involving a sampling process, which can easily introduce a selection bias (Heckman, 1977; Zadrozny, 2004). It would get even worse when the bias can reveal the label information, resulting in the “leakage features,” which are irrelevant to the content/semantic of the sentences but are predictive to the label. One ex- ample is the QuoraQP, a dataset …

Content truncated to 500 words