Negation, Coordination, and Quantifiers in Contextualized Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 3074–3085 October 12–17, 2022. 3074 Negation, Coordination, and Quantifiers in Contextualized Language Models Aikaterini-Lida Kalouli1∗, Rita Sevastjanova2∗, Christin Beck2, and Maribel Romero2 1 CIS - LMU Munich, kalouli@cis.lmu.de 2 University of Konstanz, firstname.lastname@uni.kn Abstract With the success of contextualized language models, much research explores what these models really learn and in which cases they still fail. Most of this work focuses on specific NLP tasks and on the learning outcome. Little research has attempted to decouple the mod- els’ weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. In this paper, we take up this research opportunity: based on theoretical linguistic in- sights, we explore whether the semantic con- straints of function words are learned and how the surrounding context impacts their embed- dings. We create suitable datasets, provide new insights into the inner workings of LMs vis-a- vis function words and implement an assisting visual web interface for qualitative analysis. 1 Introduction Recently, tremendous progress has been observed in the development of contextualized language models (LM). After the introduction of contextu- alized embeddings like ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), earlier static word embeddings (Mikolov et al., 2013; Pennington et al., 2014) have been sidelined, and new stan- dards have been set for the state-of-the-art. Particu- larly, LMs have been shown to learn task-agnostic properties of language (e.g., Belinkov, 2018; Con- neau et al., 2018; Peters et al., 2018) and linguistic properties that imitate the classical NLP pipeline (Tenney et al., 2019a). Despite this success, LMs cannot be taken to understand language the way humans do, as they fail to generalize on unseen data and tasks requiring compositionality (McCoy et al., 2019; Richardson et al., 2020). Research efforts concentrate on shedding light on what these models learn and consequently how they can be improved. One strand of work relies on probing/diagnostic tasks, where classifiers are ∗Equal contribution. trained on LMs to determine whether they can en- code specific linguistic properties, e.g., Marvin and Linzen (2018); Tenney et al. (2019b); Hewitt and Manning (2019); Talmor et al. (2020). Another strand of research focuses on creating adversarial test sets with hard linguistic phenomena to observe where the models fail and thus reverse-engineer them, e.g., McCoy et al. (2019); Richardson et al. (2020); Nie et al. (2020). Both strands of research have approached their goals by employing specific tasks, e.g., Natural Language Inference, Question- Answering, etc., and this has revealed weaknesses of the current models, e.g., that negation is not treated according to its semantic nature. Never- theless, there is still little work, e.g., Ethayarajh (2019); Sevastjanova et al. (2021), decoupling the weaknesses of LMs from specific NLP tasks and focusing on which of their inner workings are re- sponsible for these weaknesses. In this paper, we attempt to fill this gap by taking a closer look at function words, i.e., words with little lexical meaning. Function words have been …

Content truncated to 500 words