mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models Ryokan Ri1,2∗ ryo0123@ousia.jp Ikuya Yamada1,3 ikuya@ousia.jp Yoshimasa Tsuruoka2 tsuruoka@logos.t.u-tokyo.ac.jp 1Studio Ousia, Tokyo, Japan 2The University of Tokyo, Tokyo, Japan 3RIKEN AIP, Tokyo, Japan Abstract Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment infor- mation from Wikipedia entities. However, ex- isting methods only exploit entity information in pretraining and do not explicitly use enti- ties in downstream tasks. In this study, we explore the effectiveness of leveraging entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages with entity representations and show the model consistently outperforms word-based pretrained models in various cross- lingual transfer tasks. We also analyze the model and the key insight is that incorporat- ing entity representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a multilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits cor- rect factual knowledge more likely than using only word representations. Our source code and pretrained models are available at https: //github.com/studio-ousia/luke. 1 Introduction Pretrained language models have become crucial for achieving state-of-the-art performance in mod- ern natural language processing. In particular, mul- tilingual language models (Conneau and Lample, 2019; Conneau et al., 2020a; Doddapaneni et al., 2021) have attracted considerable attention particu- larly due to their utility in cross-lingual transfer. In zero-shot cross-lingual transfer, a pretrained encoder is fine-tuned in a single resource-rich lan- guage (typically English), and then evaluated on other languages never seen during fine-tuning. A key to solving cross-lingual transfer tasks is to ob- tain representations that generalize well across lan- guages. Several studies aim to improve multilin- gual models with cross-lingual supervision such as ∗ Work done as an intern at Studio Ousia. bilingual word dictionaries (Conneau et al., 2020b) or parallel sentences (Conneau and Lample, 2019). Another source of such information is the cross- lingual mappings of Wikipedia entities (articles). Wikipedia entities are aligned across languages via inter-language links and the text contains numer- ous entity annotations (hyperlinks). With these data, models can learn cross-lingual correspon- dence such as the words Tokyo (English) and 東 京 (Japanese) refers to the same entity. Wikipedia entity annotations have been shown to provide rich cross-lingual alignment information to improve multilingual language models (Calixto et al., 2021; Jiang et al., 2022). However, previous studies only incorporate entity information through an auxiliary loss function during pretraining, and the models do not explicitly have entity representations used for downstream tasks. In this study, we investigate the effectiveness of entity representations in multilingual language models. Entity representations are known to en- hance language models in mono-lingual settings (Zhang et al., 2019; Peters et al., 2019; Wang et al., 2021; Xiong et al., 2020; Yamada et al., 2020) presumably by introducing real-world knowledge. We show that using entity representations facili- tates cross-lingual transfer by providing language- independent features. To this end, we present a multilingual extension of LUKE …

Content truncated to 500 words