Verb Knowledge Injection for Multilingual Event Processing

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6952–6969 August 1–6, 2021. ©2021 Association for Computational Linguistics 6952 Verb Knowledge Injection for Multilingual Event Processing Olga Majewska1 Ivan Vuli´c1 Goran Glavaš2 Edoardo M. Ponti1,3 Anna Korhonen1 1Language Technology Lab, TAL, University of Cambridge, UK 2 Data and Web Science Group, University of Mannheim, Germany 3 Mila – Quebec AI Institute, Montreal, Canada 1{om304,iv250,ep490,alk23}@cam.ac.uk 2goran@informatik.uni-mannheim.de Abstract Linguistic probing of pretrained Transformer- based language models (LMs) revealed that they encode a range of syntactic and semantic properties of a language. However, they are still prone to fall back on superficial cues and simple heuristics to solve downstream tasks, rather than leverage deeper linguistic informa- tion. In this paper, we target a specific facet of linguistic knowledge, the interplay between verb meaning and argument structure. We in- vestigate whether injecting explicit informa- tion on verbs’ semantic-syntactic behaviour improves the performance of pretrained LMs in event extraction tasks, where accurate verb processing is paramount. Concretely, we im- part the verb knowledge from curated lexi- cal resources into dedicated adapter modules (verb adapters), allowing it to complement, in downstream tasks, the language knowledge ob- tained during LM-pretraining. We first demon- strate that injecting verb knowledge leads to performance gains in English event extraction. We then explore the utility of verb adapters for event extraction in other languages: we investi- gate 1) zero-shot language transfer with multi- lingual Transformers and 2) transfer via (noisy automatic) translation of English verb-based lexical knowledge. Our results show that the benefits of verb knowledge injection indeed ex- tend to other languages, even when relying on noisily translated lexical knowledge. 1 Introduction Large Transformer-based encoders, pretrained with self-supervised language modeling (LM) objec- tives, form the backbone of state-of-the-art models for most NLP tasks (Devlin et al., 2019; Yang et al., 2019b; Liu et al., 2019). Recent probes showed that they implicitly extract a non-negligible amount of linguistic knowledge from text corpora in an unsupervised fashion (Hewitt and Manning, 2019; Vuli´c et al., 2020; Rogers et al., 2020, inter alia). In downstream tasks, however, they often rely on spurious correlations and superficial cues (Niven and Kao, 2019) rather than a deep understanding of language meaning (Bender and Koller, 2020), which is detrimental to both generalisation and in- terpretability (McCoy et al., 2019). In this work, we focus on a specific facet of lin- guistic knowledge: reasoning about events.1 Iden- tifying tokens in the text that mention events and classifying the temporal and causal relations among them is crucial to understand the structure of a story or dialogue (Carlson et al., 2002; Miltsakaki et al., 2004) and to ground a text in real-world facts. Verbs (with their arguments) are prominently used for expressing events (with their participants). Thus, fine-grained knowledge about verbs, e.g., the syntactic patterns in which they partake and the semantic frames, may help pretrained encoders to achieve a deeper understanding of text and improve their …

Content truncated to 500 words