Sorting through the noise: Testing robustness of information processing in pre-trained language models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1583–1596 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1583 Sorting through the noise: Testing robustness of information processing in pre-trained language models Lalchand Pandia lcpandia@gmail.com Allyson Ettinger Department of Linguistics University of Chicago aettinger@uchicago.edu Abstract Pre-trained LMs have shown impressive per- formance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to process- ing, retaining, and applying information pre- sented in their input. In this paper we tackle a component of this question by examining ro- bustness of models’ ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systemati- cally manipulate the nature of these distractors, to shed light on dynamics of models’ use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of dis- tracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to fac- tors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by ro- bust representations of context meaning. 1 Introduction In recent years, pre-trained language models (LMs) have taken NLP by storm. As we work to inter- pret the impressive performance of these models, a persistent question is the extent to which LMs are doing something like language “understanding”. Do these models robustly extract the nuances of information conveyed in text, or is their strong per- formance driven by more superficial mechanisms? In this paper we focus on examining a particular aspect of “understanding” in pre-trained LMs: how robustly these models process, retain, and apply new facts presented in their inputs. We assume that a fundamental aspect of language “understanding” will be the capacity to represent and accumulate information from the meaning of the input text. So if we input “Sebastian lives in France, and Rowan lives in Indonesia.”, we would expect a model that understands language to form (or update) repre- sentations for these imaginary entities Sebastian and Rowan, such that those representations contain the information that Sebastian lives in France, and that Rowan lives in Indonesia. While recent work has studied models as knowledge bases, testing their ability to reproduce facts encountered during training, in this paper we are asking an importantly different question—not about what models mem- orize during training, but about how sophisticated they are in processing and representing information from new input text after training. To test LMs’ ability to process and retain infor- mation from context, we design cloze tasks that in- corporate a piece of critical information in context, and then prompt the model to …

Content truncated to 500 words