RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7077–7087 August 1–6, 2021. ©2021 Association for Computational Linguistics 7077 RAW-C: Relatedness of Ambiguous Words—in Context (A New Lexical Resource for English) Sean Trott University of California, San Diego sttrott@ucsd.edu Benjamin Bergen University of California, San Diego bkbergen@ucsd.edu Abstract Most words are ambiguous—they convey dis- tinct meanings in different contexts—and even the meanings of unambiguous words are context-dependent. Both phenomena present a challenge for NLP. Recently, the advent of con- textualized word embeddings has led to suc- cess on tasks involving lexical ambiguity, such as Word Sense Disambiguation. However, there are few tasks that directly evaluate how well these embeddings accommodate the con- tinuous, dynamic nature of word meaning— particularly in a way that matches human in- tuitions. We introduce RAW-C, a dataset of graded, human relatedness judgments for 112 ambiguous words in context (with 672 sentence pairs total), as well as human es- timates of sense dominance. The average inter-annotator agreement for the relatedness norms (assessed using a leave-one-annotator- out method) was 0.79. We then show that a measure of cosine distance, computed using contextualized embeddings from BERT and ELMo, correlates with human judgments, but that cosine distance also systematically under- estimates how similar humans find uses of the same sense of a word to be, and systematically overestimates how similar humans find uses of different-sense homonyms. Finally, we pro- pose a synthesis between psycholinguistic the- ories of the mental lexicon and computational models of lexical semantics. 1 Introduction Words mean different things in different contexts. Sometimes these meanings appear to be distinct, a phenomenon known as lexical ambiguity. In En- glish, approximately 7% of wordforms are homony- mous, i.e., they have multiple, unrelated meanings1 (e.g., “tree bark” vs. “dog bark”), and as many 1Dautriche (2015) estimates the average rate of homonymy across languages to be 4%. as 84% of wordforms are polysemous, i.e., they have multiple, related meanings (e.g., “pet chicken” vs. “roast chicken”) (Rodd et al., 2004). But even unambiguous words evoke subtly different inter- pretations depending on the context of use, i.e., their meanings are dynamic and context-dependent (Yee and Thompson-Schill, 2016; Li and Joanisse, 2021). While the uses of runs in “the boy runs” vs. “the cheetah runs” may not be considered distinct meanings, a human comprehender will likely ac- tivate a different mental image when processing each sentence (Elman, 2009). These facts present a challenge for computa- tional models of lexical semantics. Any down- stream task that involves meaning requires models capable of disambiguating among the multiple pos- sible meanings of an ambiguous word in a given context. Further, the graded nature of human se- mantic representations can influence how compre- henders construe events and participants in those events (Elman, 2009; Li and Joanisse, 2021). In turn, a number of Natural Language Processing (NLP) tasks could benefit from context-sensitive representations that go beyond discrete sense rep- resentations and capture the manner …

Content truncated to 500 words