Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3250–3258 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3250 Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models Nora Kassner∗, Philipp Dufter∗, Hinrich Sch¨utze Center for Information and Language Processing (CIS), LMU Munich, Germany {kassner,philipp}@cis.lmu.de Abstract Recently, it has been found that monolin- gual English language models can be used as knowledge bases. Instead of structural knowl- edge base queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. We translate the established bench- marks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multi- lingual knowledge base? Most prior work only considers English. Extending research to mul- tiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better perfor- mance? We find that using mBERT as a knowl- edge base yields varying performance across languages and pooling predictions across lan- guages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin. 1 Introduction Pretrained language models (LMs) (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019) can be finetuned to a variety of natural language processing (NLP) tasks and generally yield high performance. Increasingly, these models and their generative variants are used to solve tasks by sim- ple text generation, without any finetuning (Brown et al., 2020). This motivated research on how much knowledge is contained in LMs: Petroni et al. (2019) used models pretrained with masked lan- guage to answer fill-in-the-blank templates such as “Paris is the capital of [MASK].” ∗ Equal contribution - random order. Query Two most frequent predictions en X was created in MASK. [Japan (170), Italy (56), . . . ] de X wurde in MASK erstellt. [Deutschland (217), Japan (70), . . . ] it X `e stato creato in MASK. [Italia (167), Giappone (92), . . . ] nl X is gemaakt in MASK. [Nederland (172), Itali¨e (50), . . . ] en X has the position of MASK. [bishop (468), God (68), ...] de X hat die Position MASK. [WW (261), Ratsherr (108), ...] it X ha la posizione di MASK. [pastore ( 289), papa (138), ...] nl X heeft de positie van MASK. [burgemeester (400), bisschop (276) , ...] Table 1: Language bias when querying (TyQ) mBERT. Top: For an Italian cloze question, Italy is favored as country of origin. Bottom: There is no overlap be- tween the top-ranked predictions, demonstrating the in- fluence of language – even though the facts are the same: the same set of triples is evaluated across lan- guages. Table 3 shows that pooling predictions across languages addresses bias …

Content truncated to 500 words