Factual Consistency of Multilingual Pretrained Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 3046 - 3052 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Factual Consistency of Multilingual Pretrained Language Models Constanza Fierro Anders Søgaard University of Copenhagen {c.fierro,soegaard}@di.ku.dk, Abstract Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and re- cent work has shown that monolingual English language models lack consistency when predict- ing factual knowledge, that is, they fill-in-the- blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, MPARAREL1, and investi- gate (i) whether multilingual language models such as mBERT and XLM-R are more consis- tent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsis- tent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages. 1 Introduction Pretrained Language Models (PLMs) enable high- quality sentence and document representations (Pe- ters et al., 2018; Devlin et al., 2019; Yang et al., 2019; Raffel et al., 2020) and encode world knowl- edge that can be useful for downstream tasks, e.g. closed-book QA (Roberts et al., 2020), and com- monsense reasoning (Zellers et al., 2019; Talmor et al., 2019), to name a few. Recent work has used language models as knowledge bases (Petroni et al., 2019; Kassner et al., 2021a; Roberts et al., 2020) and as the basis of neural databases (Thorne et al., 2021). Such usage of PLMs relies on the assumption that we can generally trust the world knowledge that is induced from these models. Consistency is a core quality that we would like models to have when we use their stored factual knowledge. We want models to behave consistently 1https://github.com/coastalcph/mpararel on semantically equivalent inputs (Elazar et al., 2021), and to be consistent in their believes (Kass- ner et al., 2021b). Moreover we want them to be fair across languages or in other words to exhibit a consistent behaviour across languages (Choud- hury and Deshpande, 2021). Nonetheless, recent work on consistency in PLMs has shown that mod- els are brittle in their predictions when faced to irrelevant changes in the input (Gan and Ng, 2019; Ribeiro et al., 2020; Elazar et al., 2021; Ravichan- der et al., 2020). These works only considered English PLMs, while Jang et al. (2021) studied the consistency of Korean PLMs. There are, to the best of our knowledge, no resources available to measure the consistency of multilingual PLMs. Contributions In this paper, we present MPARA- REL, a multilingual version of the PARAREL dataset (Elazar et al., 2021), which we construct by automatically translating the English data to 45 languages and performing a human review of 11 of these. We then evaluate how consistent mBERT …

Content truncated to 500 words