Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6878 - 6890 December 7-11, 2022 ©2022 Association for Computational Linguistics Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples Linlin Liu1,2∗ Xin Li1 Ruidan He1 Lidong Bing1† Shafiq Joty2,3 Luo Si1 1DAMO Academy, Alibaba Group 2Nanyang Technological University, Singapore 3Salesforce Research 1{linlin.liu, xinting.lx, ruidan.he, l.bing, luo.si}@alibaba-inc.com 2srjoty@ntu.edu.sg Abstract Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks. How- ever, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretrain- ing. They often train LMs with KGs in indi- rect ways, relying on extra entity/relation em- beddings to facilitate knowledge injection. In this work, we explore methods to make bet- ter use of the multilingual annotation and lan- guage agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained directly on the knowledge triples. We first generate a large amount of multilingual synthetic sentences us- ing the Wikidata KG triples. Then based on the intra- and inter-sentence structures of the generated data, we design pretraining tasks to enable the LMs to not only memorize the factual knowledge but also learn useful logi- cal patterns. Our pretrained KMLMs demon- strate significant performance improvements on a wide range of knowledge-intensive cross- lingual tasks, including named entity recogni- tion (NER), factual knowledge retrieval, rela- tion classification, and a newly designed logi- cal reasoning task.1 1 Introduction Pretrained Language Models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have achieved superior performances on a wide range of NLP tasks. Existing PLMs usu- ally learn universal language representations from general-purpose large-scale corpora but do not con- centrate on capturing world’s factual knowledge. It has been shown that knowledge graphs (KGs), such ∗ Linlin Liu is under the Joint PhD Program between Alibaba and Nanyang Technological University. †Corresponding author. 1Our code, data and pretrained models are available at https://github.com/ntunlp/kmlm.git. as Wikidata (Vrandeˇci´c and Krötzsch, 2014) and Freebase (Bollacker et al., 2008), can provide rich factual information for better language understand- ing. Many studies have demonstrated the effec- tiveness of incorporating such factual knowledge into monolingual PLMs (Peters et al., 2019; Zhang et al., 2019; Liu et al., 2020a; Poerner et al., 2020; Wang et al., 2021a). Following this, a few recent attempts have been made to enhance multilingual PLMs with Wikipedia or KG triples (Calixto et al., 2021; Ri et al., 2022; Jiang et al., 2022). However, due to the structural difference between KG and texts, existing KG based pretraining often relies on extra relation/entity embeddings or additional KG encoders for knowledge enhancement. These ex- tra embeddings/components may add significantly more parameters which in turn increase inference complexity, or cause inconsistency between pre- train and downstream tasks. For example, mLUKE (Ri et al., 2022) has to enumerate all possible en- tity spans for NER to minimize the inconsistency caused by entity and entity position embeddings. Other methods (Liu et al., …

Content truncated to 500 words