A Structural Probe for Finding Syntax in Word Representations

Proceedings of NAACL-HLT 2019, pages 4129–4138 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4129 A Structural Probe for Finding Syntax in Word Representations John Hewitt Stanford University johnhew@stanford.edu Christopher D. Manning Stanford University manning@stanford.edu Abstract Recent work has improved our ability to detect linguistic knowledge in word repre- sentations. However, current methods for detecting syntactic knowledge do not test whether syntax trees are represented in their entirety. In this work, we propose a structural probe, which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. The probe identifies a linear transformation under which squared L2 distance encodes the distance between words in the parse tree, and one in which squared L2 norm encodes depth in the parse tree. Using our probe, we show that such transformations exist for both ELMo and BERT but not in baselines, providing evidence that entire syntax trees are embedded implicitly in deep models’ vector geometry. 1 Introduction As pretrained deep models that build contextual- ized representations of language continue to pro- vide gains on NLP benchmarks, understanding what they learn is increasingly important. To this end, probing methods are designed to evaluate the extent to which representations of language en- code particular knowledge of interest, like part-of- speech (Belinkov et al., 2017), morphology (Peters et al., 2018a), or sentence length (Adi et al., 2017). Such methods work by specifying a probe (Con- neau et al., 2018; Hupkes et al., 2018), a supervised model for finding information in a representation. Of particular interest, both for linguistics and for building better models, is whether deep models’ representations encode syntax (Linzen, 2018). Despite recent work (Kuncoro et al., 2018; Peters et al., 2018b; Tenney et al., 2019), open questions remain as to whether deep contextual models encode entire parse trees in their word representations. In this work, we propose a structural probe, a simple model which tests whether syntax trees are consistently embedded in a linear transformation of a neural network’s word representation space. Tree structure is embedded if the transformed space has the property that squared L2 distance between two words’ vectors corresponds to the number of edges between the words in the parse tree. To re- construct edge directions, we hypothesize a linear transformation under which the squared L2 norm corresponds to the depth of the word in the parse tree. Our probe uses supervision to find the trans- formations under which these properties are best approximated for each model. If such transfor- mations exist, they define inner products on the original space under which squared distances and norms encode syntax trees – even though the mod- els being probed were never given trees as input or supervised to reconstruct them. This is a structural property of the word representation space, akin to vector offsets encoding word analogies (Mikolov et al., 2013). Using our probe, we conduct a tar- geted case study, showing that ELMo (Peters et al., 2018a) and BERT (Devlin …

Content truncated to 500 words