Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3391–3401 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3391 Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings Christos Xypolopoulos∗ ´Ecole Polytechnique Antoine J.-P. Tixier* ´Ecole Polytechnique Michalis Vazirgiannis ´Ecole Polytechnique & AUEB Abstract The number of senses of a given word, or poly- semy, is a very subjective notion, which varies widely across annotators and resources. We propose a novel method to estimate polysemy based on simple geometry in the contextual embedding space. Our approach is fully un- supervised and purely data-driven. Through rigorous experiments, we show that our rank- ings are well correlated, with strong statis- tical significance, with 6 different rankings derived from famous human-constructed re- sources such as WordNet, OntoNotes, Oxford, Wikipedia, etc., for 6 different standard met- rics. We also visualize and analyze the correla- tion between the human rankings and make in- teresting observations. A valuable by-product of our method is the ability to sample, at no ex- tra cost, sentences containing different senses of a given word. Finally, the fully unsuper- vised nature of our approach makes it applica- ble to any language. Code and data are pub- licly available1. 1 Introduction Polysemy, the number of senses that a word has, is a very subjective notion, subject to individual biases. Word sense annotation has always been one of the tasks with the lowest values of inter- annotator agreement (Artstein and Poesio, 2008). Yet, creating high-quality, consistent word sense inventories is a critical pre-requisite to successful word sense disambiguation. Towards creating word sense inventories, it can be helpful to have some reliable information about polysemy. That is, knowing which words have ∗ Equal contribution. CX handled the data, generated the rankings, and sampled the examples of section 7. AJPT computed the results, plots, and wrote the paper. Both authors participated in the design of the study. 1https://github.com/ksipos/polysemy-assessment many senses and which words have only a few senses. Such information can help in creating new inventories but also in validating and interpreting existing ones. It can also help select which words to include in a study (e.g., only highly polysemous words). We propose a novel, fully unsupervised, and data-driven approach to quantify polysemy, based on basic geometry in the contextual embedding space. Contextual word embeddings have emerged in the last few years as part of the NLP transfer learning revolution. Now, entire deep models are pre-trained on huge amounts of unannotated data and fine-tuned on much smaller annotated datasets. Some of the most famous examples include ULM- FiT (Howard and Ruder, 2018) and ELMo (Pe- ters et al., 2018), both based on recurrent neu- ral networks; and GPT (Radford et al., 2018) and BERT (Devlin et al., 2018), based on transform- ers (Vaswani et al., 2017). These models all are very deep language models. During pre-training on large-scale corpora, they learn to generate powerful internal representations, including fine-grained con- textual word embeddings. For instance, in …

Content truncated to 500 words