SelfMix: Robust Learning against Textual Label Noise with Self-Mixup Training

Proceedings of the 29th International Conference on Computational Linguistics, pages 78–93 October 12–17, 2022. 78 Machine Reading, Fast and Slow: When Do Models “Understand” Language? Sagnik Ray Choudhury1*, 2, Anna Rogers2, and Isabelle Augenstein2 1University of Michigan 2University of Copenhagen sagnikrayc@gmail.com, arogers@sodas.ku.dk, augenstein@di.ku.dk Abstract Two of the most fundamental challenges in Nat- ural Language Understanding (NLU) at present are: (a) how to establish whether deep learning- based models score highly on NLU bench- marks for the ‘right’ reasons; and (b) to un- derstand what those reasons would even be. We investigate the behavior of reading compre- hension models with respect to two linguistic ‘skills’: coreference resolution and compari- son. We propose a definition for the reasoning steps expected from a system that would be ‘reading slowly’, and compare that with the be- havior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the sys- tems based on larger encoders are more likely to rely on the ‘right’ information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison. 1 Introduction Generally, human decisions may be based on deliberate, careful reasoning (‘slow thinking’) or quick heuristics (‘fast thinking’) (Kahneman, 2011). These two processes have parallels in the realm of reading comprehension (RC): a human reader would ideally fully process the text to an- swer questions, but in practice, we may deliberately skim rather than read to save effort. Even capable students may be misled by superficial cues (Acker- man et al., 2013). The previous generations of NLP models have already achieved high performance on many RC benchmarks, but they were found to often ‘read fast’, i.e. rely on shallow patterns (Chen et al., 2016; Jia and Liang, 2017; Rychalska et al., 2018). Fine-tuned Transformer-based models (De- vlin et al., 2019) still have similar shortcomings * Work done while employed at the University of Copen- hagen Context: Leo Strauss was a political philosopher and classicist. He was born in Germany ... Thoughts on Machiavelli is a book by Leo Strauss ... Question: Where was the author of Thoughts of Machiavelli born ? Answer: Germany Figure 1: A sample question from the SQuAD (Rajpurkar et al., 2016) dataset. green tokens are the words that a reader relying on coreference resolution would take into account, and red tokens are the words that could be used to answer the question with entity type matching. (Sugawara et al., 2020; Rogers et al., 2020; Sen and Saffari, 2020; Kassner and Schütze, 2020, inter alia) in RC, as well as other tasks (McCoy et al., 2019; Jin et al., 2020). Consider the example in Figure 1. A human reader would ideally construct the coreference chain resolving the pronoun ‘he’ to ‘Leo Strauss’. A possible heuristic-based solution is entity type matching (Jia and Liang, 2017): a model could observe that a ‘where’ question can only be an- swered by a ‘location’ and …

Content truncated to 500 words