Does BERT Recognize an Agent? Modeling Dowty’s Proto-Roles with Contextual Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 4101–4112 October 12–17, 2022. 4101 Does BERT Recognize an Agent? Modeling Dowty’s Proto-Roles with Contextual Embeddings Mattia Proietti CoLing Lab, University of Pisa m.proietti2@studenti.unipi.it Gianluca E. Lebani Ca’ Foscari University of Venice gianluca.lebani@unive.it Alessandro Lenci CoLing Lab, University of Pisa alessandro.lenci@unipi.it Abstract Contextual embeddings build multidimensional representations of word tokens based on their context of occurrence. Such models have been shown to achieve a state-of-the-art performance on a wide variety of tasks. Yet, the commu- nity struggles in understanding what kind of semantic knowledge these representations en- code. We report a series of experiments aimed at investigating to what extent one of such mod- els, BERT, is able to infer the semantic relations that, according to Dowty’s Proto-Roles theory, a verbal argument receives by virtue of its role in the event described by the verb. This hypothesis were put to test by learning a linear mapping from the BERT’s verb em- beddings to an interpretable space of seman- tic properties built from the linguistic dataset by White et al. (2016). In a first experiment we tested whether the semantic properties in- ferred from a typed version of the BERT em- beddings would be more linguistically plausi- ble than those produced by relying on static embeddings. We then move to evaluate the se- mantic properties inferred from the contextual embeddings both against those available in the original dataset, as well as by assessing their ability to model the semantic properties pos- sessed by the agent of the verbs participating in the so-called causative alternation. 1 Introduction In the last two decades, word embeddings have become one of the most widely used tools for the encoding of lexical meaning in computational mod- els of language. Different flavours of such models have been proposed, all of which have in common the idea of representing lexical elements as multi- dimensional vectors inferred from their context of occurrence (for a review, see Lenci, 2018). The last wave of word embeddings followed the transformer-based models breakthrough (Vaswani et al., 2017), that resulted in the development of the so-called contextual embeddings. These represen- tations are generated by models like BERT (Devlin et al., 2019) or GPTs (Radford and Narasimhan, 2018; Radford et al., 2019) and derive their name by their ability to keep track of the different con- texts in which a word occurs, giving different vec- tor representations for the same word appearing with different surrounding neighbours (for review, see Liu et al., 2020; Ethayarajh, 2019). This has been a major improvement over static embeddings obtained from models such as LSA (Landauer and Dumais, 1997), GloVe (Pennington et al., 2014) and Word2Vec (Mikolov et al., 2013), allowing this kind of representation to reach state-of-the-art performance in a great variety of Natural Language Processing (NLP) tasks. Notwithstanding their wide usage, mainly due to their great empirical successes, the community still struggles to understand what kind of informa- tion word embeddings are actually able to encode about language …

Content truncated to 500 words