Evaluating language models for the retrieval and categorization of lexical collocations

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1406–1417 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1406 Evaluating language models for the retrieval and categorization of lexical collocations Luis Espinosa-Anke1, Joan Codina-Filb`a2, Leo Wanner3,2 1School of Computer Science and Informatics, Cardiff University, UK 2TALN Research Group, Pompeu Fabra University, Barcelona, Spain 3Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain espinosa-ankel@cardiff.ac.uk, joan.codina|leo.wanner@upf.edu Abstract Lexical collocations are idiosyncratic com- binations of two syntactically bound lexical items (e.g., “heavy rain”, “take a step” or “undergo surgery”). Understanding their de- gree of compositionality and idiosyncrasy, as well their underlying semantics, is crucial for language learners, lexicographers and down- stream NLP applications alike. In this paper we analyse a suite of language models for col- location understanding. We first construct a dataset of apparitions of lexical collocations in context, categorized into 16 representative semantic categories. Then, we perform two experiments: (1) unsupervised collocate re- trieval, and (2) supervised collocation classi- fication in context. We find that most models perform well in distinguishing light verb con- structions, especially if the collocation’s first argument acts as a subject, but often fail to distinguish, first, different syntactic structures within the same semantic category, and second, finer-grained categories which restrict the set of correct collocates1. 1 Introduction Language models (LMs) such as BERT (Devlin et al., 2018), and its variants SpanBERT (Joshi et al., 2020), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), etc. have proven extremely flexi- ble, as they behave as unsupervised multitask learn- ers (Radford et al., 2019), and can be leveraged in a wide array of NLP tasks almost out-of-the- box (see, e.g., the GLUE and SuperGLUE results in Wang et al. (2019b) and Wang et al. (2019a), respectively). They have also been harnessed as supporting resources for knowledge-based NLP (Petroni et al., 2019), as they capture a wealth of 1The resources associated with this paper are available at https://github.com/luisespinosaanke/ lexicalcollocations. linguistic phenomena (Rogers et al., 2020). Re- cently, a great deal of research analyzed the degree to which they encode, e.g., morphological (Edmis- ton, 2020), syntactic (Hewitt and Manning, 2019), or lexico-semantic structures (Joshi et al., 2020). However, less work explored so far how LMs in- terpret phraseological units at various degrees of compositionality. This is crucial for understanding the suitability of different text representations (e.g., static vs contextualized word embeddings) for en- coding different types of multiword expressions (Shwartz and Dagan, 2019), which, in turn, can be useful for extracting latent world or commonsense information (Zellers et al., 2018). One central type of phraselogical units are lexical collocations, defined as restricted co- occurrences of two syntactically bound lexical items (Kilgarriff, 2006), such that one of the items (the base) conditions the selection of the other item (the collocate) to express a specific meaning. For instance, the base lecture conditions the collocates give or deliver to express the meaning ‘perform’, the base applause conditions the selection of the collocate …

Content truncated to 500 words