Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339–2352 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2339 Syntactic Data Augmentation Increases Robustness to Inference Heuristics Junghyun Min1 R. Thomas McCoy1 Dipanjan Das2 Emily Pitler2 Tal Linzen1 1Department of Cognitive Science, Johns Hopkins University, Baltimore, MD 2Google Research, New York, NY {jmin10, tom.mccoy, tal.linzen}@jhu.edu {dipanjand, epitler}@google.com Abstract Pretrained neural models such as BERT, when fine-tuned to perform natural language infer- ence (NLI), often show high accuracy on stan- dard datasets, but display a surprising lack of sensitivity to word order on controlled chal- lenge sets. We hypothesize that this issue is not primarily caused by the pretrained model’s limitations, but rather by the paucity of crowd- sourced NLI examples that might convey the importance of syntactic structure at the fine- tuning stage. We explore several methods to augment standard training sets with syntacti- cally informative examples, generated by ap- plying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inver- sion, improved BERT’s accuracy on controlled examples that diagnose sensitivity to word or- der from 0.28 to 0.73, without affecting per- formance on the MNLI test set. This improve- ment generalized beyond the particular con- struction used for data augmentation, suggest- ing that augmentation causes BERT to recruit abstract syntactic representations. 1 Introduction In the supervised learning paradigm common in NLP, a large collection of labeled examples of a particular classification task is randomly split into a training set and a test set. The system is trained on this training set, and is then evaluated on the test set. Neural networks—in particular systems pretrained on a word prediction objective, such as ELMo (Peters et al., 2018) or BERT (Devlin et al., 2019)—excel in this paradigm: with large enough pretraining corpora, these models match or even exceed the accuracy of untrained human annotators on many test sets (Raffel et al., 2019). At the same time, there is mounting evidence that high accuracy on a test set drawn from the same distribution as the training set does not indi- cate that the model has mastered the task. This dis- crepancy can manifest as a sharp drop in accuracy when the model is applied to a different dataset that illustrates the same task (Talmor and Berant, 2019; Yogatama et al., 2019), or as excessive sensitivity to linguistically irrelevant perturbations of the input (Jia and Liang, 2017; Wallace et al., 2019). One such discrepancy, where strong perfor- mance on a standard test set did not correspond to mastery of the task as a human would define it, was documented by McCoy et al. (2019b) for the Natural Language Inference (NLI) task. In this task, the system is given two sentences, and is expected to determine whether one (the premise) entails the other (the hypothesis). Most if not all humans would agree that NLI requires sensitivity to syntactic structure; for example, the following sentences do not entail each other, even though they contain the …

Content truncated to 500 words