How is BERT surprised? Layerwise detection of linguistic anomalies

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4215–4228 August 1–6, 2021. ©2021 Association for Computational Linguistics 4215 How is BERT surprised? Layerwise detection of linguistic anomalies Bai Li1,4, Zining Zhu1,4 Guillaume Thomas2, Yang Xu1,3,4, Frank Rudzicz1,4,5 1 University of Toronto, Department of Computer Science 2 University of Toronto, Department of Linguistics 3 University of Toronto, Cognitive Science Program 4 Vector Institute for Artificial Intelligence 5 Unity Health Toronto {bai, zining, yangxu, frank}@cs.toronto.edu guillaume.thomas@utoronto.ca Abstract Transformer language models have shown re- markable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermedi- ate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judge- ment benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is mor- phosyntactic than when it is semantic, while commonsense anomalies do not exhibit sur- prisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of lin- guistic anomalies. 1 Introduction Transformer-based language models (LMs) have achieved remarkable success in numerous natural language processing tasks, prompting many prob- ing studies to determine the extent of their linguis- tic knowledge. A popular approach is to formulate the problem as a multiple-choice task, where the LM is considered correct if it assigns higher like- lihood to the appropriate word than an inappro- priate one, given context (Gulordava et al., 2018; Ettinger, 2020; Warstadt et al., 2020). The like- lihood score, however, only gives a scalar value of the degree that a word is anomalous in context, and cannot distinguish between different ways that a word might be anomalous. It has been proposed that there are differ- ent types of linguistic anomalies. Chomsky The cat won 't eating the food 0 1 2 3 4 5 6 7 8 9 10 11 12 Layer The plane laughed at the runway 0 1 2 3 4 5 6 7 8 9 10 11 12 Figure 1: Example sentence with a morphosyntactic anomaly (left) and semantic anomaly (right) (anoma- lies in bold). Darker colours indicate higher surprisal. We investigate several patterns: first, surprisal at lower layers corresponds to infrequent tokens, but this effect diminishes towards upper layers. Second, morphosyn- tactic violations begin to trigger high surprisals at an earlier layer than semantic violations. (1957) distinguished semantic anomalies (“color- less green ideas sleep furiously”) from ungram- maticality (“furiously sleep ideas green color- less”). Psycholinguistic studies initially suggested that different event-related potentials (ERPs) are produced in the brain depending on the type of anomaly; e.g., semantic anomalies produce nega- tive …

Content truncated to 500 words