PolyLM: Learning about Polysemy through Language Modeling

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 563–574 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 563 PolyLM: Learning about Polysemy through Language Modeling Alan Ansell1,2 , Felipe Bravo-Marquez3, Bernhard Pfahringer2 1Language Technology Lab, University of Cambridge 2Department of Computer Science, University of Waikato 3Department of Computer Science, University of Chile & IMFD aja63@cam.ac.uk, fbravo@dcc.uchile.cl, bernhard@waikato.ac.nz Abstract To avoid the “meaning conflation deficiency” of word embeddings, a number of models have aimed to embed individual word senses. These methods at one time performed well on tasks such as word sense induction (WSI), but they have since been overtaken by task-specific techniques which exploit contextualized em- beddings. However, sense embeddings and contextualization need not be mutually exclu- sive. We introduce PolyLM, a method which formulates the task of learning sense embed- dings as a language modeling problem, allow- ing contextualization techniques to be applied. PolyLM is based on two underlying assump- tions about word senses: firstly, that the prob- ability of a word occurring in a given con- text is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausi- ble in the context than the others. We evalu- ate PolyLM on WSI, showing that it performs considerably better than previous sense embed- ding techniques, and matches the current state- of-the-art specialized WSI method despite hav- ing six times fewer parameters. Code and pre-trained models are available at https:// github.com/AlanAnsell/PolyLM. 1 Introduction Much work in NLP has been dedicated to vector representations of words, but it has been recognized since as early as (Sch¨utze, 1998) that such repre- sentations fail to capture the polysemous nature of many words, conflating their multiple senses into a single point in semantic space. There have been several attempts at embedding individual word senses to avoid this issue, termed the “meaning conflation deficiency” by Camacho-Collados and Pilehvar (2018) in their survey on the area. We propose PolyLM, an unsupervised sense em- bedding model which is effective and easy to apply to downstream tasks. PolyLM can be thought of as both a (masked) language model and a sense model, as it calculates a probability distribution both over words and word senses at masked positions. The formulation is derived from two observations about word senses: firstly, that the probability of a word occurring in a given context is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausible in the context than the others. There are several reasons for the interest in sense representations. The first is the downsides associ- ated with the meaning conflation deficiency. Word embedding models can have difficulty distinguish- ing which sense of an ambiguous word applies in a given context (Yaghoobzadeh and Sch¨utze, 2016). Additionally, homonymy and polysemy …

Content truncated to 500 words