Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3130–3148 July 9-14, 2023 ©2023 Association for Computational Linguistics Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis Mario Giulianelli◁, Iris Luden◁, Raquel Fernández◁, Andrey Kutuzov⋄ ◁University of Amsterdam ⋄University of Oslo m.giulianelli@uva.nl, irisluden@gmail.com, raquel.fernandez@uva.nl, andreku@ifi.uio.no Abstract We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations. Given a collection of usage examples for a target word, and the corre- sponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label. We demon- strate how the resulting sense labels can make existing approaches to semantic change analy- sis more interpretable, and how they can allow users—historical linguists, lexicographers, or social scientists—to explore and intuitively explain diachronic trajectories of word mean- ing. Semantic change analysis is only one of many possible applications of the ‘definitions as representations’ paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embed- dings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP. 1 Introduction Accurate semantic understanding in language tech- nologies is typically powered by distributional word representations and pre-trained language mod- els (LMs). Due to their subsymbolic nature, how- ever, such methods lack in explainability and in- terpretability, leading to insufficient trust in end users. An example application which requires capturing word meaning with its nuanced context- determined modulations is lexical semantic change analysis, a task which consists in detecting whether a word’s meaning has changed over time, for example by acquiring or losing a sense. Mod- ern semantic change detection systems rely on static and contextualised word representations, LM- based lexical replacement, grammatical profiles, supervised word sense and word-in-context disam- biguation (Kutuzov et al., 2018; Tahmasebi et al., 2021). But the main potential end users of these technologies—historical linguists, lexicographers, and social scientists—are still somewhat reluctant to adopt them precisely because of their lack of ex- planatory power. Lexicographers, for instance, are not satisfied with detecting that a word has or hasn’t changed its meaning over the last ten years; they want descriptions of old and new senses in human- readable form, possibly accompanied by additional layers of explanation, e.g., specifying the type of semantic change (such as broadening, narrowing, and metaphorisation) the word has undergone. Our work is an attempt to bridge the gap be- tween computational tools for semantic understand- ing and their users. We propose to replace black- box contextualised token embeddings produced by large LMs with a new type of interpretable lex- ical semantic representation: automatically gen- erated contextualised word definitions (Gardner et al., 2022). In this paradigm, the usage of the word ‘apple’ in the sentence ‘She tasted a fresh green apple’ is represented not with a dense high- dimensional vector but with the context-dependent …

Content truncated to 500 words