When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 227–244 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 227 When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships Gabriella Chronis Department of Linguistics The University of Texas at Austin Austin, TX 78705 USA gabriellachronis@utexas.edu Katrin Erk Department of Linguistics The University of Texas at Austin Austin, TX 78705 USA katrin.erk@utexas.edu Abstract This paper investigates contextual language models, which produce token representa- attions, as a resource for lexical semantics the word or type level. We con- struct multi-prototype word embeddings from bert-base-uncased (Devlin et al., 2018). These embeddings retain contextual knowl- edge that is critical for some type-level tasks, while being less cumbersome and less sub- ject to outlier effects than exemplar models. Similarity and relatedness estimation, both type-level tasks, benefit from this contextual knowledge, indicating the context-sensitivity of these processes. BERT’s token level knowl- edge also allows the testing of a type-level hypothesis about lexical abstractness, demon- strating the relationship between token-level behavior and type-level concreteness ratings. Our findings provide important insight into the interpretability of BERT: layer 7 approximates semantic similarity, while the final layer (11) approximates relatedness. 1 Introduction The rampant success enjoyed by contextualized lan- guage models (CLMs) like CoVe (McCann et al., 2017), ElMo (Peters et al., 2018), BERT (Devlin et al., 2018), and RoBERTa (Liu et al., 2019b) has precipitated a deluge of research into analyzing and interpreting their functionality. But to date, there has been little work analyzing their lexical seman- tic knowledge. This paper seeks to answer two questions: 1) Is it possible to generate useful static word-type embeddings from BERT activations for word tokens? 2) What sort of semantic relations are represented in embeddings generated from BERT? ‘Useful’ word embeddings are those which suc- cessfully represent target semantic relations and enable the testing of linguistic and cognitive hy- potheses. Many linguistic tasks and questions con- cern word meanings at the type level—that is, what a word means in general, abstracted away from any particular context. Similarity, relatedness, and ab- stractness are often construed as properties at the type-level: most similarity and relatedness datasets contain judgments on isolated word pairs, and ab- stractness datasets contain judgments on isolated words. CLMs produce representations at the token level: the vector representation of a word varies depend- ing on its context of occurrence. How can contex- tual representations be helpful for type-level tasks? Similarity and relatedness are context-sensitive pro- cesses. For example, the relevant features of water used for calculating its similarity to land are dif- ferent from those for calculating its similarity to coffee. BERT’s contextual knowledge is useful for dealing with the effects of this variation on simi- larity and relatedness judgments. Relative abstract- ness/concreteness is another property often treated as a type-level phenomenon. However, this prop- erty may be detectable through token-level interac- tions. We hypothesize that a high degree of con- textual variation may be an indicator …

Content truncated to 500 words