The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10859 - 10882 December 7-11, 2022 ©2022 Association for Computational Linguistics The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative Leonie Weissweiler*⋄, Valentin Hofmann†*, Abdullatif Köksal*⋄, Hinrich Schütze*⋄ *Center for Information and Language Processing, LMU Munich ⋄Munich Center of Machine Learning †Faculty of Linguistics, University of Oxford {weissweiler,akoksal}@cis.lmu.de valentin.hofmann@ling-phil.ox.ac.uk Abstract Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the con- nection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of dif- ferent granularity that combine syntax and se- mantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of- the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most com- monly studied constructions, the English com- parative correlative (CC). We conduct exper- iments examining the classification accuracy of a syntactic probe on the one hand and the models’ behaviour in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning. While human-like perform- ance of PLMs on many NLP tasks has been al- leged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge. 1 Introduction The sentence “The better your syntax, the better your semantics.” contains a construction called the English comparative correlative (CC; Fillmore, 1986). Paraphrased, it could be read as “If your syntax is better, your semantics will also be better.” Humans reading this sentence are capable of doing two things: (i) recognising that two instances of “the” followed by an adjective/adverb in the compar- ative as well as a phrase of the given structure (i.e., the syntax of the CC) express a specific meaning (i.e., the semantics of the CC); (ii) understanding the semantic meaning conveyed by the CC, i.e., understanding that in a sentence of the given struc- ture, the second half is somehow correlated with the first. In this paper, we ask the following question: are pretrained language models (PLMs) able to achieve these two steps? This question is important for two reasons. Firstly, we hope that recognising the CC and understanding its meaning is challenging for PLMs, helping to set the research agenda for further improvements. Secondly, the CC is one of the most commonly studied constructions in construction grammar (CxG), a usage-based syntax paradigm from cognitive linguistics, thus providing an interesting alternative to the currently prevailing practice of analysing the syntactic capabilities of PLMs with theories from generative grammar (e.g., Marvin and Linzen, 2018). We divide our investigation into two parts. In the first part, we examine the CC’s syntactic prop- erties and how they are represented by PLMs, …

Content truncated to 500 words