Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3651–3661 June 6–11, 2021. ©2021 Association for Computational Linguistics 3651 Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks Iacer Calixto1,2 Alessandro Raganato3 Tommaso Pasini4,∗ 1Center for Data Science, New York University 2ILLC, University of Amsterdam 3Department of Digital Humanities, University of Helsinki, Finland 4Department of Computer Science, University of Copenhagen iacer.calixto@nyu.edu, alessandro.raganato@helsinki.fi, tommaso.pasini@di.ku.dk Abstract Masked language models have quickly be- come the de facto standard when processing text. Recently, several approaches have been proposed to further enrich word representa- tions with external knowledge sources such as knowledge graphs. However, these models are devised and evaluated in a monolingual setting only. In this work, we propose a language- independent entity prediction task as an in- termediate training procedure to ground word representations on entity semantics and bridge the gap across different languages by means of a shared vocabulary of entities. We show that our approach effectively injects new lexical- semantic knowledge into neural models, im- proving their performance on different seman- tic tasks in the zero-shot crosslingual setting. As an additional advantage, our intermediate training does not require any supplementary in- put, allowing our models to be applied to new datasets right away. In our experiments, we use Wikipedia articles in up to 100 languages and already observe consistent gains compared to strong baselines when predicting entities using only the English Wikipedia. Further adding extra languages lead to improvements in most tasks up to a certain point, but overall we found it non-trivial to scale improvements in model transferability by training on ever in- creasing amounts of Wikipedia languages. 1 Introduction Pretrained Multilingual Masked Language Models (MMLMs) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and their variants have achieved state-of-the-art results across diverse natural language understanding tasks. Typically, a MMLM model is pretrained on very large amounts of raw text in different languages using the masked language modelling (MLM) objective and is further finetuned on (usually limited amounts of) task data. ∗Work carried out while at the University of Rome “La Sapienza”. In the zero-shot crosslingual setting, which is our focus in this paper, a MMLM is finetuned on the target task using data in a single language (e.g., English) and is evaluated on the same task but in different languages (e.g., non-English languages). We introduce the multilingual Wikipedia hy- perlink prediction objective to contextualise words in a text with entities and concepts from an external knowledge source by using Wikipedia articles in up to 100 languages. Hyperlink predic- tion is a knowledge-rich task designed to (1) inject semantic knowledge from Wikipedia entities and concepts into the MMLM token representations, and (2) with a similar motivation as the translated language modelling loss of Conneau and Lample (2019), i.e., to inject explicit language-independent knowledge into a model trained via self-supervised learning, but in our case without parallel data. We devise a …

Content truncated to 500 words