Competency Problems: On Finding and Removing Artifacts in Language Data

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1801–1813 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1801 Competency Problems: On Finding and Removing Artifacts in Language Data Matt Gardner*∗♠ William Merrill*♠ Jesse Dodge♠ Matthew E. Peters♠ Alexis Ross♠ Sameer Singh♥ Noah A. Smith♣♠ ♠Allen Institute for Artificial Intelligence ♥University of California, Irvine ♣University of Washington Abstract Much recent work in NLP has documented dataset artifacts, bias, and spurious correla- tions between input features and output la- bels. However, how to tell which features have “spurious” instead of legitimate corre- lations is typically left unspecified. In this work we argue that for complex language un- derstanding tasks, all simple feature correla- tions are spurious, and we formalize this no- tion into a class of problems which we call competency problems. For example, the word “amazing” on its own should not give informa- tion about a sentiment label independent of the context in which it appears, which could in- clude negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that re- alistic datasets will increasingly deviate from competency problems as dataset size increases. This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases. Our theoretical treatment of this prob- lem also allows us to analyze proposed solu- tions, such as making local edits to dataset in- stances, and to give recommendations for fu- ture data collection and model design efforts that target competency problems. 1 Introduction Attempts by the natural language processing com- munity to get machines to understand language or read text are often stymied in part by issues in our datasets (Chen et al., 2016; Sugawara et al., 2018). Many recent papers have shown that pop- ular datasets are prone to shortcuts, dataset arti- facts, bias, and spurious correlations (Jia and Liang, 2017; Rudinger et al., 2018; Costa-jussà et al., 2019). While these empirical demonstrations of ∗Equal contribution 102 103 104 105 n 0.0 0.2 0.4 0.6 0.8 1.0 p(y xi) competitionn vacationn firstn sleepingc noc nobodyc catc catsc outdoorse persone peoplee outsidee animale Artifact statistics in SNLI = 0.01/28k neutral contradict entailment Figure 1: A statistical test for deviation from a com- petency problem, where no individual feature (here words) should give information about the class la- bel, plotting the number of occurrences of each word against the conditional probability of the label given the presence of the word. The label associated with each point is marked by color and superscript. All fea- tures above the blue line have detectable correlation with class labels, using a very conservative Bonferroni- corrected statistical test. deficiencies in the data are useful, they often leave unanswered fundamental questions of what exactly makes a correlation “spurious”, instead of a feature that is legitimately predictive of some target …

Content truncated to 500 words