Spotting Spurious Data with Neural Networks

Proceedings of NAACL-HLT 2018, pages 2006–2016 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Spotting Spurious Data with Neural Networks Hadi Amiri, Timothy A. Miller, Guergana Savova Boston Children’s Hospital Informatics Program, Harvard Medical School {firstname.lastname}@childrens.harvard.edu Abstract Automatic identification of spurious instances (those with potentially wrong labels datasets) can improve the quality of exist-in ing language resources, especially when an- notations are obtained through crowdsourc- ing or automatically generated based on coded rankings. In this paper, we present an ef- fective approach inspired by queueing the- ory and psychology of learning to automati- cally identify spurious instances in datasets. Our approach discriminates instances based on their “difficulty to learn,” determined by a downstream learner. Our method can be ap- plied to any dataset assuming the existence of a neural network model for the target task of the dataset. Our best approach outper- forms competing state-of-the-art baselines and has a MAP of 0.85 and 0.22 in identifying spurious instances in synthetic and carefully- crowdsourced real-world datasets respectively. 1 Introduction The importance of error-free language resources cannot be overstated as errors can inversely af- fect interpretations of the data, models developed from the data, and decisions made based on the data. Although the quality of language resources can be improved through good annotation guide- lines, test questions, etc., annotation noise still exists (Gupta et al., 2012; Lasecki et al., 2013). For example, Figure 1 shows sample spurious in- stances (those with potentially wrong labels) in CIFAR-10 (Krizhevsky, 2009) which is a bench- mark dataset for object classification. Spurious instances can mislead systems, and, if available in test data, lead to unrealistic comparison among competing systems. Previous works either directly identify noise in datasets (Hovy et al., 2013; Dickinson and Meurers, 2003; Eskin, 2000; Loftsson, 2009), (a) Truck (b) Airplane (c) Cat Figure 1: Spurious instances in CIFAR-10. (a) not a truck, (b) missing annotation for car as another object category, (c) incomplete image of a cat. or develop models that are more robust against noise (Guan et al., 2017; Natarajan et al., 2013; Zhu et al., 2003; Zhu and Wu, 2004). Furthermore, recent works on adversarial perturbation have tackled this problem (Goodfellow et al., 2015; Feinman et al., 2017). However, most previous approaches require either annotations generated by each individual annotator (Guan et al., 2017), or both task-specific and instance-type (genuine or adversarial) labels for training (Hendrik Met- zen et al., 2017; Zheng et al., 2016), or noise-free data (Xiao et al., 2015). Such information is often not available in the final release of most datasets. Current approaches utilize prediction probabil- ity/loss of instances to tackle the above challenges in identifying spurious instances. This is because prediction probability/loss of spurious instances tend to be lower than that of genuine instances (He and Garcia, 2009). In particular, the Bayesian Uncertainty model (Feinman et al., 2017) defines spurious instances as those that have greater un- certainty (variance) in their stochastic predictions, and the Variational Inference model (Rehbein and Ruppenhofer, …

Content truncated to 500 words