Who did what to Whom? Language models and humans respond diversely to features affecting argument hierarchy construction

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 254–265 November 20–23, 2022. ©2022 Association for Computational Linguistics 254 Who did what to Whom? Language models and humans respond diversely to features affecting argument hierarchy construction Xiaonan Xu University of Cologne, Germany Haoshuo Chen Nokia Bell Labs, USA Abstract Pre-trained transformer-based language models have achieved state-of-the-art performance in many areas of NLP. It is still an open question whether the models are capable of integrating syntax and semantics in language processing like humans. This paper investigates if mod- els and humans construct argument hierarchy similarly with the effects from telicity, agency, and individuation, using the Chinese structure “NP1+BA/BEI+NP2+VP”. We present both hu- mans and six transformer-based models with prepared sentences and analyze their preference between BA (view NP1 as an agent) and BEI (NP2 as an agent). It is found that the models and humans respond to (non-)agentive features in telic context and atelic feature very similarly. However, the models show insufficient sensi- tivity to both pragmatic function in expressing undesirable events and different individuation degrees represented by human common nouns vs. proper names. By contrast, humans rely heavily on these cues to establish the thematic relation between two arguments NP1 and NP2. Furthermore, the models tend to interpret the subject as an agent, which is not the case for humans who align agents independently of sub- ject position in Mandarin Chinese.1 1 Introduction Pre-trained transformer-based language models (LMs) keep achieving state-of-the-art performance in NLP tasks. Many studies have indicated that pre-trained LMs can learn syntactic knowledge (e.g., Linzen et al. 2016; Gulordava et al. 2018 for subject-verb agreement, Wilcox et al. 2018 for filler-gap dependencies, Futrell et al. 2019 for garden-path effects) and semantic knowledge (e.g., Zhao et al. 2021 for telicity , Kementchedjhieva et al. 2021 for causality bias, Misra et al. 2020 for semantic priming, Misra et al. 2021 for typicality, 1Dataset for both humans and language models, and analysis code are available at https://github.com/ NLPbelllabs/WhoWhom.git Ettinger 2020 for role reversal and same-category distinctions). However, to what extent LMs can acquire knowledge in the syntax-semantics inter- face is still an open question. To answer this ques- tion, we explore arguments hierarchy construction which identifies the thematic roles of arguments in the semantic domain and aligns arguments and subject/object in the syntactic domain. In this hi- erarchy, the active, controlling agent (prototyp- ical actor) outranks the affected patient (proto- typical undergoer), i.e., who did what to whom? (Van Valin Jr, 1990; Van Valin and LaPolla, 1997; Bornkessel et al., 2005). The mapping between thematic roles (agent/patient) and syntactic struc- ture (subject/object) varies depending on various features. In this paper, we investigate whether pre-trained transformer-based LMs and humans behave simi- larly in the argument hierarchy construction using the Chinese structure “NP1+BA/BEI+NP2+VP”. This structure provides a unique opportunity to examine the alignment through the occurrence of BA/BEI (Deng et …

Content truncated to 500 words