Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15354–15373 July 9-14, 2023 ©2023 Association for Computational Linguistics Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning Barun Patra∗, Saksham Singhal∗, Shaohan Huang∗, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song Microsoft {bapatra, saksingh, shaohanh, v-zewenchi, lidong1, fuwei, vchaudhary, xiaso}@microsoft.com Abstract In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state- of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical ap- plications. We show that going beyond English- centric bitexts, coupled with a novel sampling strategy aimed at reducing under-utilization of training data, substantially boosts perfor- mance across model sizes for both Electra and MLM pre-training objectives. We introduce XY-LENT: X-Y bitext enhanced Language ENcodings using Transformers which not only achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands, is also competitive across bands. Our XY- LENTXL variant outperforms XLM-RXXL and exhibits competitive performance with mT5XXL while being 5x and 6x smaller respectively. We then show that our proposed method helps ame- liorate the curse of multilinguality, with the XY- LENTXL achieving 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band. We then analyze our models performance on extremely low resource languages and posit that scaling alone may not be sufficient for im- proving the performance in this scenario. 1 Introduction Recent advancements in Natural Language Pro- cessing (NLP) have been a direct consequence of leveraging foundational models (Bommasani et al., 2021), pretrained on a large text corpora in a self- supervised fashion. This has also been the case for multilingual NLP where pre-trained models like multilingual BERT (mBERT) (Devlin, 2018; Devlin et al., 2019), XLM (Conneau and Lample, 2019), XLM-Roberta (Conneau et al., 2020), XLM- Electra (Chi et al., 2022) and mT5 (Xue et al., 2021) ∗Equal Contribution. 228 229 230 231 232 233 234 75 76 77 78 79 80 81 82 83 84 85 86 B B B B L L L L XL XL XL XL XXL XXL # Parameters XNLI Accuracy XYLENT XLM-E XLM-R mT5 Figure 1: The proposed XY-LENT model (green line) achieves SoTA performance within all band sizes and is competitive performance across larger model-size bands. The parameter efficiency of XY-LENTXL partic- ularly stands out, outperforming XLM-RXXL and being competitive with mT5XXL while being 5x and 6x smaller than them respectively. We also present the performance of XLM-E which used as a baseline in this paper. have all shown non-trivial performance gains, es- pecially in the setup of zero-shot transfer, and have been the work-horse for a diverse number of mul- tilingual tasks. Given their ubiquitous applicabil- ity in zero-shot downstream scenarios, improving the quality and enabling their usage in resource- constrained applications is also an important vein of research which we explore in this paper. A source of improvement for …

Content truncated to 500 words