Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 149–164 July 13-14, 2023 ©2023 Association for Computational Linguistics Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity Lindia Tjuatja, Emmy Liu, Lori Levin, Graham Neubig Language Technologies Institute Carnegie Mellon University {ltjuatja, mengyan3, lsl, gneubig}@cs.cmu.edu Abstract Recent advances in large language models have prompted researchers to examine their abilities across a variety of linguistic tasks, but little has been done to investigate how models han- dle the interactions in meaning across words and larger syntactic forms—i.e. phenomena at the intersection of syntax and semantics. We present the semantic notion of agentivity as a case study for probing such interactions. We created a novel evaluation dataset by utilitiz- ing the unique linguistic properties of a sub- set of optionally transitive English verbs. This dataset was used to prompt varying sizes of three model classes to see if they are sensitive to agentivity at the lexical level, and if they can appropriately employ these word-level pri- ors given a specific syntactic context. Over- all, GPT-3 text-davinci-003 performs extremely well across all experiments, outper- forming all other models tested by far. In fact, the results are even better correlated with hu- man judgements than both syntactic and seman- tic corpus statistics. This suggests that LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and dis- covery than select corpora for certain tasks. 1 Introduction Consider the English sentences in (1) below: (1) a. This author writes easily. b. This passage writes easily. These sentences display an interesting property of certain optionally transitive verbs in English. Al- though they share an identical surface syntactic structure—a noun phrase in subject position fol- lowed by the intransitive form of the verb and an adverb phrase modifying the verb—they entail very different things about the roles of their subjects. The subject of (1a) is someone that does the action of writing; in other words, this author is an agent in the writing event. On the other hand, the subject of (1b), this passage, doesn’t do any writing—it is what is created in the event of writ- ing. In contrast to this author, this passage is a patient. The agent and patient roles are not dis- crete categories, but rather prototypes on opposite ends of a continuum. These “protoroles” have a number of contributing properties such as causing an event for agents and undergoing change of state for patients (Dowty, 1991). The contrast between the minimal pair in (1) suggests that there are lexical semantic properties of the subjects that give rise to these two distinct readings: one that describes how the subject gen- erally does an action as in (1a), and another that describes how an event generally unfolds when the subject undergoes an action as in (1b). Intuitively, a speaker may know from the meaning of author that authors are animate, have some degree of voli- tion, and typically write …

Content truncated to 500 words