X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5943–5959, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5943 X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models Zhengbao Jiang†, Antonios Anastasopoulos♣,∗, Jun Araki‡, Haibo Ding‡, Graham Neubig† †Languages Technologies Institute, Carnegie Mellon University ♣Department of Computer Science, George Mason University ‡Bosch Research Abstract Language models (LMs) have proven surpris- ingly successful at capturing factual knowl- edge by completing cloze-style fill-in-the- blank questions such as “Punta Cana is lo- cated in _.” However, while knowledge is both written and queried in many languages, studies on LMs’ factual representation ability have almost invariably been performed on En- glish. To assess factual knowledge retrieval in LMs in different languages, we create a mul- tilingual benchmark of cloze-style probes for 23 typologically diverse languages. To prop- erly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algo- rithms to generate multi-token predictions. Ex- tensive experimental results provide insights about how well (or poorly) current state-of-the- art LMs perform at this task in languages with more or fewer available resources. We fur- ther propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effective- ness on several benchmark languages. Bench- mark data and code have be released at https: //x-factr.github.io. 1 Introduction Language models (LMs; (Church, 1988; Kneser and Ney, 1995; Bengio et al., 2003)) learn to model the probability distribution of text, and in doing so capture information about various aspects of the syntax or semantics of the language at hand. Recent works have presented intriguing results demonstrat- ing that modern large-scale LMs also capture a significant amount of factual knowledge (Petroni et al., 2019; Jiang et al., 2020; Poerner et al., 2019). This knowledge is generally probed by having the LM fill in the blanks of cloze-style prompts such as ∗: Work done at Carnegie Mellon University. The first two authors contributed equally. en fr nl ru es jp vi zh hu ko tr he 6.1 2.2 2 1.6 1.5 1.2 1.2 1.1 0.5 0.5 0.4 0.3 Wikipedia Size (in million articles) el war mr mg bn tl sw pa ceb yo ilo 0.2 0.2 0.1 0.09 0.09 0.07 0.06 0.04 0.03 0.03 0.02 fact ⟨Bloomberg L.P., founded_in, New York⟩ en prompt [X] was founded in [Y]. es prompt [X] fue [fundar.Gerund;X] en [Y]. ↓ ↓ ↓ es sentence Bloomberg L.P. fue fundada en ⟨mask⟩ ×1 ∼ 5. es outputs prediction #tokens confidence 2012 1 -1.90 Nueva York 2 -0.61 EE. UU 3 -1.82 Chicago, Estados Unidos 4 -3.58 2012 Bloomberg L.P 5 -3.06 Figure 1: X-FACTR contains 23 languages, for which the data availability varies dramatically. Prompts get instantiated to produce grammatical sentences with dif- ferent numbers of mask tokens and are used to ob- tain predictions for [Y]. In this Spanish example, the verb gerund “fundar” to found is rendered as “fun- dada” to agree in gender and number with the subject …

Content truncated to 500 words