Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8517–8532 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8517 Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution Denis Emelin1 Rico Sennrich2,1 1School of Informatics, University of Edinburgh 3Department of Computational Linguistics, University of Zurich D.Emelin@sms.ed.ac.uk, sennrich@cl.uzh.ch Abstract Winograd schemas are a well-established tool for evaluating coreference resolution (CoR) and commonsense reasoning (CSR) capabili- ties of computational models. So far, schemas remained largely confined to English, limiting their utility in multilingual settings. This work presents Wino-X, a parallel dataset of Ger- man, French, and Russian schemas, aligned with their English counterparts. We use this resource to investigate whether neural ma- chine translation (NMT) models can perform CoR that requires commonsense knowledge and whether multilingual language models (MLLMs) are capable of CSR across multi- ple languages. Our findings show Wino-X to be exceptionally challenging for NMT sys- tems that are prone to undesireable biases and unable to detect disambiguating information. We quantify biases using established statisti- cal methods and define ways to address both of these issues. We furthermore present evi- dence of active cross-lingual knowledge trans- fer in MLLMs, whereby fine-tuning models on English schemas yields CSR improvements in other languages.1 1 Introduction Originally introduced in (Winograd, 1972), Wino- grad schemas (schemas from here on) have be- come an established tool for probing the ability of computational models to reason about natural language. Either viewed through the lens of coref- erence (CoR) as in (Levesque et al., 2012) or, more recently, framed as a gap-filling task (Sakaguchi et al., 2020), schemas are assumed to require com- monsense knowledge to be resolved correctly. Consider the following schema: The trophy doesn’t fit into the brown suitcase because it is too [large / small]. Here, the pronoun it has two possible antecedents (trophy / suitcase), with the 1The Wino-X dataset and experimental codebase can be found at https://github.com/demelin/Wino-X. Figure 1: Wino-X examples. Cross-lingual samples are used to evaluate translation models, whereas multilin- gual instances are compatible with MLLMs. Corefer- ent words are highlighted with the same color, while disambiguating trigger words are underlined. choice of the antecedent determined by the trigger word (large / small). To connect the pronoun to its true antecedent, a model must ’know’ that objects that are too large cannot fit into containers and that containers that are too small cannot house objects. When translating an instance of a schema (i.e. the schema with a fixed trigger word) into lan- guages such as German, where pronouns and their antecedents must agree in their grammatical gen- der, translation models must implicitly perform the CoR step to produce accurate translations. A competent translation model is, therefore, expected to identify the correct antecedent as reflected by the target pronoun choice. In the first part of this work, we construct cross-lingual instances by align- ing English instances with their translations into morphologically rich languages, so as to probe the 8518 robustness of CoR in current NMT models, as …

Content truncated to 500 words