Attentional Probe: Estimating a Module’s Functional Potential

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11459 - 11472 December 7-11, 2022 ©2022 Association for Computational Linguistics The Architectural Bottleneck Principle Tiago Pimentel∗, Josef Valvoda∗, Niklas Stoehr Ryan Cotterell University of Cambridge ETH Zürich {tp472, jv406}@cam.ac.uk {niklas.stoehr, ryan.cotterell}@inf.ethz.ch Abstract In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model’s representations contain. This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component. Relying on this principle, we estimate how much syntactic information is available to transformers through our attentional probe, a probe that exactly resembles a transformer’s self-attention head. Experimentally, we find that, in three models (BERT, ALBERT, and RoBERTa), a sentence’s syntax tree is mostly extractable by our probe, suggesting these models have access to syntactic information while composing their contextual representations. Whether this information is actually used by these models, however, remains an open question. https://github.com/rycolab/ attentional-probe 1 Introduction The surprising performance of pretrained language models on diverse natural language processing tasks has sparked interest in their analysis. Probing is one of the most prevalent methods employed to engage in such an analysis. In a typical probing study (Alain and Bengio, 2016; Belinkov et al., 2017; Adi et al., 2017, inter alia), the weights of the model under consideration are first frozen. A probe is then trained on top of the model’s contextual representations in an attempt to predict one of the input sentence’s properties, e.g., its syntactic parse. Unfortunately, best practices on how to design such probes remain contested. On one side of the debate, some argue for simplicity, suggesting that simple probes are to be ∗Equal contribution. preferred so that we can distinguish probing from simply learning an NLP task (Hewitt and Liang, 2019). On the other side of the debate, some argue we need complex probes in order to extract all relevant information from the representations (Saphra and Lopez, 2019; Pimentel et al., 2020b). Bridging the gap, some have also called for a compromise, advocating that all probes on the complexity–accuracy Pareto curve should be considered (Pimentel et al., 2020a). In this paper, we propose the architectural bottleneck principle (ABP) as a guideline for constructing useful probes. Under the ABP, a probe’s architecture should mirror a component of the model being probed. Previous work has mostly focused on how much information is contained in a set of representations. However, if we care about whether the information is in fact used by the model, we should instead ask how much information the model in question could use.1 Under this perspective, the probed model’s architecture acts as a natural bottleneck to how much information the model could use—and should thus also act …

Content truncated to 500 words