Analysing Lexical Semantic Change with Contextualised Word Representations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960–3973 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3960 Analysing Lexical Semantic Change with Contextualised Word Representations Mario Giulianelli Marco Del Tredici Raquel Fern´andez Institute for Logic, Language and Computation University of Amsterdam {m.giulianelli|m.deltredici|raquel.fernandez}@uva.nl Abstract This paper presents the first unsupervised ap- proach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain repre- sentations of word usages, clusters these rep- resentations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method cap- tures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction. 1 Introduction In the fourteenth century the words boy and girl referred respectively to a male servant and a young person of either sex (Oxford English Dictionary). By the fifteenth century a narrower usage had emerged for girl, designating exclusively female in- dividuals, whereas by the sixteenth century boy had lost its servile connotation and was more broadly used to refer to any male child, becoming the mas- culine counterpart of girl (Bybee, 2015). Word meaning is indeed in constant mutation and, since correct understanding of the meaning of individual words underpins general machine reading compre- hension, it has become increasingly relevant for computational linguists to detect and characterise lexical semantic change—e.g., in the form of laws of semantic change (Dubossarsky et al., 2015; Xu and Kemp, 2015; Hamilton et al., 2016)—with the aid of quantitative and reproducible evaluation pro- cedures (Schlechtweg et al., 2018). Most recent studies have focused on shift de- tection, the task of deciding whether and to what extent the concept evoked by a word has changed between time periods (e.g., Gulordava and Ba- roni, 2011; Kim et al., 2014; Kulkarni et al., 2015; Del Tredici et al., 2019; Hamilton et al., 2016; Bam- ler and Mandt, 2017; Rosenfeld and Erk, 2018). This line of work relies mainly on distributional semantic models, which produce one abstract repre- sentation for every word form. However, aggregat- ing all senses of a word into a single representation is particularly problematic for semantic change as word meaning hardly ever shifts directly from one sense to another, but rather typically goes through polysemous stages (Hopper et al., 1991). This limi- tation has motivated recent work on word sense in- duction across time periods (Lau et al., 2012; Cook et al., 2014; Mitra et al., 2014; Frermann and Lap- ata, 2016; Rudolph and Blei, 2018; Hu et al., 2019). Word senses, however, have shortcomings them- selves as they are a discretisation of word meaning, which is continuous in nature and modulated by context to convey ad-hoc interpretations (Brugman, 1988; Kilgarriff, 1997; Paradis, 2011). In this …

Content truncated to 500 words