Large Scale Substitution-based Word Sense Induction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4738 - 4752 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Large Scale Substitution-based Word Sense Induction Matan Eyal1 Shoval Sadde1 Hillel Taub-Tabib1 Yoav Goldberg1,2 1 Allen Institute for AI, Israel 2 Bar Ilan University, Ramat-Gan, Israel matane,shovals,hillelt,yoavg@allenai.org Abstract We present a word-sense induction method based on pre-trained masked language mod- els (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged accord- ing to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy. Furthermore, by training a static word embed- dings algorithm on the sense-tagged corpus, we obtain high-quality static senseful embed- dings. These outperform existing senseful em- beddings methods on the WiC dataset and on a new outlier detection dataset we developed. The data driven nature of the algorithm allows to induce corpora-specific senses, which may not appear in standard sense inventories, as we demonstrate using a case study on the scien- tific domain. 1 Introduction Word forms are ambiguous, and derive meaning from the context in which they appear. For example, the form “bass” can refer to a musical instrument, a low-frequency sound, a type of voice, or a kind of fish. The correct reference is determined by the surrounding linguistic context. Traditionally, this kind of ambiguity was dealt via word sense disam- biguation (WSD), a task that disambiguates word forms in context between symbolic sense-ids from a sense inventory such as WordNet (Miller, 1992) or, more recently, BabelNet (Navigli and Ponzetto, 2010). Such sense inventories rely heavily on man- ual curation, are labor intensive to produce, are not available in specialized domains and inherently unsuitable for words with emerging senses.1 This 1For example, in current WordNet version, Corona has 6 synsets, none of them relates to the novel Coronavirus. can be remedied by word sense induction (WSI), a task where the input is a given word-type and a corpus, and the output is a derived sense inventory for that word. Then, sense disambiguation can be performed over the WSI-derived senses. The introduction of large-scale pre-trained LMs and Masked LMs (MLM) seemingly made WSI/WSD tasks obsolete: instead of representing tokens with symbols that encode sense informa- tion, each token is associated with a contextualized vector embeddings that captures various aspects of its in-context semantics, including the word-sense. These contextualized vectors proved to be very ef- fective as features for downstream NLP tasks. How- ever, contextualized embeddings also have some major shortcomings: most notably for our case, they are expensive to store (e.g. BERT embeddings are 768 or 1024 floating point numbers for each token), and are hard to index and query at scale. Even if we do manage to store and query them, they …

Content truncated to 500 words