Lexical Semantics with Large Language Models: A Case Study of English “break”

Findings of the Association for Computational Linguistics: EACL 2023, pages 490–511 May 2-6, 2023 ©2023 Association for Computational Linguistics Lexical Semantics with Large Language Models: A Case Study of English break∗ Erika Petersen Stanford University epetsen@stanford.edu Christopher Potts Stanford University cgpotts@stanford.edu Abstract Large neural language models (LLMs) can be powerful tools for research in lexical semantics. We illustrate this potential using the English verb break, which has numerous senses and appears in a wide range of syntactic frames. We show that LLMs capture known sense dis- tinctions and can be used to identify informa- tive new sense combinations for further analy- sis. More generally, we argue that LLMs are aligned with lexical semantic theories in provid- ing high-dimensional, contextually modulated representations, but LLMs’ lack of discrete fea- tures and dependence on usage-based data of- fer a genuinely new perspective on traditional problems in lexical semantics. 1 Introduction Pater (2019) builds a compelling case that linguis- tic and neural network research have great potential for common ground and common cause. His case has only grown stronger in recent years, with the arrival of large neural language models (LLMs) that provide semantically rich, contextual represen- tations (McCann et al., 2017; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). In this paper, we argue that LLMs are power- ful devices for studying lexical semantics in ways that can deeply inform linguistic theory. We illus- trate this with a detailed case study of the lexical semantics of the English verb break, building on a richly annotated dataset from Petersen (2020) and drawing on methods from prior work in this area (Camacho-Collados and Pilehvar, 2018; Ten- ney et al., 2019; Garí Soler et al., 2019; Reif et al., 2019; Wiedemann et al., 2019; Branco et al., 2020; Nair et al., 2020; Li and Joanisse, 2021; Loureiro et al., 2021; Trott and Bergen, 2021; Apidianaki, 2022; McCrae et al., 2022). Break has long been ∗Data and code available at https://github.com/ epetsen/break-llms. central to theoretical work in lexical semantics be- cause it has a staggering range of senses that appear to be systematically related to its argument struc- ture. Our central empirical finding is that LLM representations capture many of these known sense distinctions and can be used to identify new sense combinations for further analysis. We use these findings as a chance to reflect on the core theoretical commitments of lexical seman- tics as they pertain to LLM-based investigations. Our discussion is centered around the three tenets of lexical semantics given in Table 1: lexical repre- sentations are high dimensional, contextually mod- ulated, and include discrete features. The high dimensionality property is not phrased as a direct claim in the literature as far as we know, but it reflects the practice of linguists, who iden- tify numerous interacting features of lexical items. Section 2 offers a summary picture for break. Sim- ilarly, discreteness is often assumed by linguists working in the broadly generative tradition. For our purposes, the key question is whether …

Content truncated to 500 words