Did they answer? Subjective acts and intents in conversational discourse

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 132–138 June 6–11, 2021. ©2021 Association for Computational Linguistics 132 A Non-Linear Structural Probe Jennifer C. WhiteD Tiago PimentelD Naomi Saphra@ Ryan CotterellD,Q DUniversity of Cambridge, @University of Edinburgh, QETH Zürich jw2088@cam.ac.uk, tp472@cam.ac.uk n.saphra@ed.ac.uk, ryan.cotterell@inf.ethz.ch Abstract Probes are models devised to investigate the encoding of knowledge—e.g. syntac- tic structure—in contextual representations. Probes are often designed for simplicity, which has led to restrictions on probe design that may not allow for the full exploitation of the structure of encoded information; one such restriction is linearity. We examine the case of a structural probe (Hewitt and Man- ning, 2019), which aims to investigate the en- coding of syntactic structure in contextual rep- resentations through learning only linear trans- formations. By observing that the structural probe learns a metric, we are able to kernel- ize it and develop a novel non-linear variant with an identical number of parameters. We test on 6 languages and find that the radial- basis function (RBF) kernel, in conjunction with regularization, achieves a statistically sig- nificant improvement over the baseline in all languages—implying that at least part of the syntactic knowledge is encoded non-linearly. We conclude by discussing how the RBF ker- nel resembles BERT’s self-attention layers and speculate that this resemblance leads to the RBF-based probe’s stronger performance. 1 Introduction Probing has been widely used in an effort to bet- ter understand what linguistic knowledge may be encoded in contextual word representations such as BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018). These probes tend to be designed with simplicity in mind and with the intent of revealing what linguistic structure is encoded in an embed- ding, rather than simply learning to perform an NLP task (Hewitt and Liang, 2019; Zhang and Bow- man, 2018; Voita and Titov, 2020) This preference for simplicity has often led researchers to place re- strictions on probe designs that may not allow them to fully exploit the structure in which information is encoded (Saphra and Lopez, 2019; Pimentel et al., 2020b,a). This preference has led many researchers to advocate the use of linear probes over non-linear ones (Alain and Bengio, 2017). This paper treats and expands upon the structural probe of Hewitt and Manning (2019), who crafted a custom probe with the aim of investigating the encoding of sentence syntax in contextual representations. They treat probing for syntax as a distance learning problem: they learn a linear transformation that warps the space such that two words that are syntactically close to one another (in terms of distance in a dependency tree) should have contextual representations whose Euclidean distance is small. This linear approach performs well, but the restriction to learning only linear transformations seems arbitrary. Why should it be the case that this information would be encoded linearly within the representations? In this paper, we recast Hewitt and Manning (2019)’s structural probing framework as a general metric learning …

Content truncated to 500 words