Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3363–3377 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3363 Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? Abhilasha Ravichander1 Yonatan Belinkov2∗ Eduard Hovy1 1Language Technologies Institute, Carnegie Mellon University 2Technion – Israel Institute of Technology aravicha@cs.cmu.edu belinkov@technion.ac.il, hovy@cmu.edu Abstract Although neural models have achieved impres- sive results on several NLP benchmarks, lit- tle is understood about the mechanisms they use to perform language tasks. Thus, much re- cent attention has been devoted to analyzing the sentence representations learned by neural encoders, through the lens of ‘probing’ tasks. However, to what extent was the information encoded in sentence representations, as dis- covered through a probe, actually used by the model to perform its task? In this work, we examine this probing paradigm through a case study in Natural Language Inference, show- ing that models can learn to encode linguistic properties even if they are not needed for the task on which the model was trained. We fur- ther identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments. Finally, through a set of controlled synthetic tasks, we demonstrate models can encode these prop- erties considerably above chance-level even when distributed in the data as random noise, calling into question the interpretation of abso- lute claims on probing tasks.1 1 Introduction Neural models have established state-of-the-art per- formance on several NLP benchmarks (Kim, 2014; Seo et al., 2017; Chen et al., 2017; Devlin et al., 2019). However, these models can be opaque and difficult to interpret, posing barriers to widespread adoption and deployment in safety-critical or user- facing settings (Belinkov and Glass, 2019). How can we know what information, if any, neural mod- els learn and leverage to perform a task? This ques- ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion. 1Code and data available at https://github.com/ AbhilashaRavichander/probing-probing. Sentence Representation Auxiliary Task 2. Freeze 3. Probe Model Control Dataset Sentence Representation 1. Train 2. Freeze Probe acc. Auxiliary Task 3. Probe Outcome: Similar probing accuracy for model trained on main dataset or control dataset. Model trained on main dataset Model trained on control dataset Main Dataset 1. Train Figure 1: Illustration of our control dataset methodol- ogy for evaluating probing classifiers. Control datasets are constructed such that a linguistic feature is not dis- criminative with respect to the task. Representations from models trained on the main dataset and control dataset are probed for the linguistic feature, and demon- strate similiar probing performance. tion has spurred considerable community effort to develop methods to analyze neural models, moti- vated by interest not just to have models perform tasks well, but also to understand the mechanisms by which they operate. A popular approach to model introspection is to associate the representations learned by the neural network with linguistic properties …

Content truncated to 500 words