A Tale of a Probe and a Parser

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7389–7395 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7389 A Tale of a Probe and a Parser Rowan Hall MaudslayD Josef ValvodaD Tiago PimentelD Adina Williams@ Ryan CotterellD,Q DUniversity of Cambridge @Facebook AI Research QETH Zürich rh635@cam.ac.uk, jv406@cam.ac.uk, tp472@cam.ac.uk, adinawilliams@fb.com, ryan.cotterell@inf.ethz.ch Abstract Measuring what linguistic information is encoded in neural models of language has be- come popular in NLP. Researchers approach this enterprise by training “probes”— supervised models designed to extract linguistic structure from another model’s output. One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations. The structural probe has a novel design, unattested in the parsing literature, the precise benefit of which is not immediately obvious. To explore whether syntactic probes would do better to make use of existing techniques, we compare the struc- tural probe to a more traditional parser with an identical lightweight parameterisation. The parser outperforms structural probe on UUAS in seven of nine analysed languages, often by a substantial amount (e.g. by 11.1 points in English). Under a second less common metric, however, there is the opposite trend—the struc- tural probe outperforms the parser. This begs the question: which metric should we prefer? 1 Introduction Recently, unsupervised sentence encoders such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have become popular within NLP. These pre-trained models boast impressive perfor- mance when used in many language-related tasks, but this gain has come at the cost of interpretability. A natural question to ask, then, is whether these models encode the traditional linguistic structures one might expect, such as part-of-speech tags or dependency trees. To this end, researchers have in- vested in the design of diagnostic tools commonly referred to as probes (Alain and Bengio, 2017; Conneau et al., 2018; Hupkes et al., 2018; Poliak et al., 2018; Marvin and Linzen, 2018; Niven and Kao, 2019). Probes are supervised models designed to extract a target linguistic structure from the output representation learned by another model. Based on the authors’ reading of the probing lit- erature, there is little consensus on where to draw the line between probes and models for performing a target task (e.g. a part-of-speech tagger versus a probe for identifying parts of speech). The main distinction appears to be one of researcher intent: probes are, in essence, a visualisation method (Hup- kes et al., 2018). Their goal is not to best the state of the art, but rather to indicate whether certain in- formation is readily available in a model—probes should not “dig” for information, they should just expose what is already present. Indeed, a suffi- ciently expressive probe with enough training data could learn any task (Hewitt and Liang, 2019), but this tells us nothing about a representation, so it is beside the point. For this reason, probes are made “simple” (Liu et al., 2019), which usually …

Content truncated to 500 words