How about Time? Probing a Multilingual Language Model for Temporal Relations

Proceedings of the 29th International Conference on Computational Linguistics, pages 3197–3209 October 12–17, 2022. 3197 How About Time? Probing a Multilingual Language Model for Temporal Relations Tommaso Caselli , Irene Dini , Felice Dell’Orletta CLCG, University of Groningen ItaliaNLP Lab, ILC-CNR “Antonio Zampolli” Pisa t.caselli@rug.nl, {irene.dini, felice.dellorletta}@ilc.cnr.it Abstract This paper presents a comprehensive set of probing experiments using a multilingual lan- guage model, XLM-R, for temporal relation classification between events in four languages. Results show an advantage of contextualized embeddings over static ones and a detrimen- tal role of sentence level embeddings. While obtaining competitive results against state-of- the-art systems, our probes indicate a lack of suitable encoded information to properly ad- dress this task. 1 Introduction Time is a pervasive element of human life with no counterpart in any other cognitive do- mains (Bonomi and Zucchi, 2001). Such perva- siveness is mirrored in natural languages through sets of devices that allows speakers to refer to time, to reason about time and things that unfold in time. Reasoning about time is one of the central com- ponents of common sense knowledge (Pianesi and Varzi, 1996; Boyd, 2010; Geva et al., 2021) and its modeling has been at the core of many early approaches in Computational Linguistics and Ar- tificial Intelligence (Schank and Abelson, 1975; McDermott, 1982; Allen, 1984; Passonneau, 1988; Moens and Steedman, 1988). More recently, spe- cific Natural Language Understanding (NLU) tasks related to time have been developed, ranging from the identification of temporal expressions (Mani et al., 2001; Mazur and Dale, 2010), to measur- ing the duration of events (Pan et al., 2006b,a; Zhou et al., 2019), and the ability to order them chronologically(Mani et al., 2003; UzZaman and Allen, 2010; Ning et al., 2018; Wen et al., 2021). More complex tasks have challenged models to extract storylines (Chambers and Jurafsky, 2008; Minard et al., 2015; Caselli and Inel, 2018), under- stand narratives (Mostafazadeh et al., 2017, 2020; Lal et al., 2021), and answer temporally related questions (Llorens et al., 2015; Ning et al., 2020). Recent work has focused on recasting temporal relation classification as a Natural Language In- ference (NLI) task where fine-tuned pre-trained language models (PTLMs) have achieved good re- sults (Vashishtha et al., 2020). Embedding representations, both static and con- textual, have shown to play a key role to improve systems’ results on different time-related bench- marks, especially for the classification of tempo- ral relations between pairs of events (Mirza and Tonelli, 2016; Cheng et al., 2020). When it comes to contextualized embeddings the probing of such models for temporal knowledge has not been prop- erly investigated yet. If we embrace the vision of PTLMs as large repositories of linguistic knowl- edge (Derby et al., 2021; Mosbach et al., 2020; Mi- aschi et al., 2020), it is a natural question to probe these models for their knowledge about events and time. We present an extensive study on temporal relation probing of PTLMs using five temporally annotated corpora in four languages (i.e., English, French, Spanish, and …

Content truncated to 500 words