Probing for Predicate Argument Structures in Pretrained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4622 - 4632 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Probing for Predicate Argument Structures in Pretrained Language Models Simone Conia1 and Roberto Navigli2 Sapienza NLP Group 1Department of Computer Science 2Department of Computer, Control and Management Engineering Sapienza University of Rome conia@di.uniroma1.it navigli@diag.uniroma1.it Abstract Thanks to the effectiveness and wide avail- ability of modern pretrained language models (PLMs), recently proposed approaches have achieved remarkable results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL). These results have prompted researchers to investigate the inner workings of modern PLMs with the aim of understanding how, where, and to what ex- tent they encode information about SRL. In this paper, we follow this line of research and probe for predicate argument structures in PLMs. Our study shows that PLMs do en- code semantic structures directly into the con- textualized representation of a predicate, and also provides insights into the correlation be- tween predicate senses and their structures, the degree of transferability between nominal and verbal structures, and how such structures are encoded across languages. Finally, we look at the practical implications of such insights and demonstrate the benefits of embedding pred- icate argument structure information into an SRL model. 1 Introduction Semantic Role Labeling (SRL) is often defined in- formally as the task of automatically answering the question “Who did What to Whom, Where, When and How?” (Màrquez et al., 2008) and is, there- fore, thought to be a fundamental step towards Natural Language Understanding (Navigli, 2018). Over the past few years, SRL has started to gain renewed traction, thanks mainly to the effective- ness and wide availability of modern pretrained language models (PLMs), such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) and BART (Lewis et al., 2020). Current approaches have, in- deed, attained impressive results on standard evalu- ation benchmarks for dependency- and span-based, multilingual and cross-lingual SRL (He et al., 2019; Li et al., 2019; Cai and Lapata, 2020; Conia and Navigli, 2020; Blloshmi et al., 2021; Conia et al., 2021). Despite the remarkable benefits provided by the rich contextualized word representations coming from PLMs, the novelties introduced in recent state- of-the-art models for SRL revolve primarily around developing complexities on top of such word repre- sentations, rather than investigating what happens inside a PLM. For example, the SRL systems of He et al. (2019) and Conia and Navigli (2020) take advantage only of BERT’s uppermost hidden layers to build their input word representations. However, the revolution that PLMs have sparked in numer- ous areas of Natural Language Processing (NLP) has motivated researchers in the community to in- vestigate the inner workings of such models, with the aim of understanding how, where, and to what extent they encode information about specific tasks. This research has revealed that different layers en- code significantly different features (Tenney et al., 2019; Vuli´c et al., 2020). In perhaps one of …

Content truncated to 500 words