On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3400–3413 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3400 On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks Stephen Mussmann∗ Robin Jia∗ Percy Liang Computer Science Department, Stanford University, Stanford, CA mussmann@stanford.edu {robinjia,pliang}@cs.stanford.edu Abstract Many pairwise classification tasks, such as paraphrase detection and open-domain ques- tion answering, naturally have extreme label imbalance (e.g., 99.99% of examples are neg- atives). In contrast, many recent datasets heuristically choose examples to ensure la- bel balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only 2.4% average pre- cision when evaluated on realistically imbal- anced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncer- tain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average preci- sion to 32.5% on QQP and 20.1% on WikiQA. 1 Introduction For most pairwise classification tasks in NLP, the most realistic data distribution has extreme label imbalance (e.g., 99.99% of examples have the same label). In question deduplication (Iyer et al., 2017), the vast majority of pairs of questions from an on- line forum are not duplicates. In open-domain ques- tion answering (Yang et al., 2015; Lee et al., 2019), almost any randomly sampled document will not answer a given question. Random pairs of sen- tences from a diverse distribution will have no re- lation between them in natural language inference (Bowman et al., 2015), as opposed to an entailment or contradiction relationship. While past work has recognized the impor- tance of label imbalance in NLP (Lewis et al., 2004; Chawla et al., 2004), many recently released datasets are heuristically collected to ensure label balance, generally for ease of training. For instance, ∗Authors contributed equally. Training Evaluation Heuristic, balanced dataset Query access to imbalanced all-pairs dataset Imbalanced all-pairs dataset Adaptive Framework Static Framework Heuristic, balanced dataset SOTA models yield poor performance Figure 1: Modern benchmarks often use heuristically balanced data for training and evaluation. We find that models trained on this data perform poorly on the very imbalanced all-pairs distribution and develop adaptive methods to collect training data for this setting. the Quora Question Pairs (QQP) dataset (Iyer et al., 2017) was generated by mining non-duplicate ques- tions that were heuristically determined to be near- duplicates. The SNLI dataset had crowdworkers generate inputs to match a specified label distri- bution (Bowman et al., 2015). In this work, we show that models trained on heuristically balanced datasets deal poorly with natural label imbalance at test time: They have very low average precision on realistically imbalanced test data created by taking all pairs of test utterances. Instead of heuristically producing static training datasets, we study adaptive data collection meth- ods. In particular, we apply the pool-based active learning framework (Settles, 2009) to extremely imbalanced pairwise …

Content truncated to 500 words