AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7151–7162 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7151 AM2ICO: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples Qianchu Liu1, Edoardo M. Ponti1 2, Diana McCarthy1, Ivan Vuli´c1, Anna Korhonen1 1 Language Technology Lab, TAL, University of Cambridge, UK 2 Mila/McGill University, Montreal, Canada {ql261,ep490,iv250,alk23}@cam.ac.uk diana@dianamccarthy.co.uk Abstract Capturing word meaning in context and dis- tinguishing between correspondences and vari- ations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evalu- ate lexical semantics “in-context” have vari- ous limitations. In particular, 1) their lan- guage coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pre- trained encoders, and 3) no support for cross- lingual evaluation. In order to address these gaps, we present AM2ICO (Adversarial and Multilingual Meaning in Context), a wide- coverage cross-lingual and multilingual evalu- ation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation mod- els to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challeng- ing nature of AM2ICO. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English. 1 Introduction Pretrained language models (LMs) such as BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) offer a natural way to distinguish differ- ent word meanings in context without performing explicit sense disambiguation. This property of “meaning contextualization” is typically evaluated either via standard entity linking (Rao et al., 2013; Shen et al., 2014) and Word Sense Disambiguation (WSD) tasks (Navigli, 2009; Moro et al., 2014; Raganato et al., 2017) or, recently, via the Word-in- Context (WiC) evaluation paradigm (Pilehvar and Camacho-Collados, 2019; Raganato et al., 2020). Although monolingual evaluation in English is still predominant, a need has been recognized to construct similar resources for other languages to support cross-lingual evaluation and model di- agnostics. This includes multilingual and cross- lingual WSD benchmarks (Navigli and Ponzetto, 2012; Navigli et al., 2013; Scarlini et al., 2020; Barba et al., 2020, inter alia), cross-lingual entity linking (Tsai and Roth, 2016; Raiman and Raiman, 2018; Upadhyay et al., 2018) and, most recently, multilingual WiC (termed XL-WiC) spanning 12 languages (Raganato et al., 2020). This most recent WiC evaluation approach is particularly attractive as 1) it bypasses the depen- dence on modeling predefined ontologies (entity linking) and explicit sense inventories (WSD), and 2) it is framed as a simple binary classification task: for a target word w appearing in two different con- texts c1 and c2, the system must decide whether w conveys the same meaning in both contexts, or not. However, …

Content truncated to 500 words