Prix-LM: Pretraining for Multilingual Knowledge Base Construction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5412 - 5424 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Prix-LM: Pretraining for Multilingual Knowledge Base Construction Wenxuan Zhou1∗, Fangyu Liu2∗, Ivan Vuli´c2, Nigel Collier2, Muhao Chen1 1LUKA Lab, University of Southern California, USA 2Language Technology Lab, TAL, University of Cambridge, UK {zhouwenx,muhaoche}@usc.edu {fl399,iv250,nhc30}@cam.ac.uk Abstract Knowledge bases (KBs) contain plenty of structured world and commonsense knowl- edge. As such, they often complement dis- tributional text-based information and facili- tate various downstream tasks. Since their manual construction is resource- and time- intensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enrich- ing multilingual KBs. Besides wider appli- cation, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource lan- guages can be transferred to low-resource ones. To achieve this, it is crucial to represent mul- tilingual knowledge in a shared/unified space. To this end, we propose a unified representa- tion model, Prix-LM , for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs, and tune a multilingual lan- guage encoder XLM-R via a causal language modeling objective. Prix-LM integrates use- ful multilingual and KB-based factual knowl- edge into a single model. Experiments on stan- dard entity-related tasks, such as link predic- tion in multiple languages, cross-lingual en- tity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains re- ported over strong task-specialised baselines. 1 Introduction Multilingual knowledge bases (KBs), such as DB- Pedia (Lehmann et al., 2015), Wikidata (Vrandeˇci´c and Krötzsch, 2014), and YAGO (Suchanek et al., 2007), provide structured knowledge expressed in ∗ Indicating equal contribution. Prix-LM The Tale of Genji has genre Mono gatari Mono THE TALE OF GENJI, has-genre, MONOGATARI 源⽒物語 …… Multilingual Causal LM THE TALE OF GENJI 源⽒物語, ジャンルは, ラブストーリー THE TALE OF GENJI, translation, 源⽒物語 translation 源⽒物語, ジャンルは, 王室関連の話 源⽒物語, ジャンルは, ⽂学-⼩説 Multilingual KGs Figure 1: An illustration of the main idea support- ing Prix-LM: it infuses complementary multilingual knowledge from KGs into a multilingual causal LM; e.g., Japanese KG stores more comprehensive genre information of The Tale of Genji than KGs in other languages. Through cross-lingual links (translations), such knowledge is then propagated across languages. multiple languages. Those KBs are modeled as knowledge graphs (KGs) that possess two types of knowledge: monolingual triples which describe relations of entities, and cross-lingual links which match entities across languages. The knowledge stored in such KGs facilitates various downstream applications such as question answering (Dai et al., 2016; Bauer et al., 2018; Wang et al., 2021b), rec- ommendation (Zhang et al., 2016; Wang et al., 2018, 2021c), and dialogue systems (Madotto et al., 2018; Liu et al., 2019; Yang et al., 2020). Manually constructing large-scale …

Content truncated to 500 words