Cross-lingual Continual Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3908–3943 July 9-14, 2023 ©2023 Association for Computational Linguistics Cross-lingual Continual Learning Meryem M’hamdi Xiang Ren Information Sciences InstituteJonathan May University of Southern California {meryem, xiangren, jonmay}@isi.edu Abstract The longstanding goal of multi-lingual learning has been to develop a universal cross-lingual model that can withstand the changes in multi- lingual data distributions. There has been a large amount of work to adapt such multi- lingual models to unseen target languages. However, the majority of work in this direction focuses on the standard one-hop transfer learn- ing pipeline from source to target languages, whereas in realistic scenarios, new languages can be incorporated at any time in a sequential manner. In this paper, we present a principled Cross-lingual Continual Learning (CCL) evalu- ation paradigm, where we analyze different cat- egories of approaches used to continually adapt to emerging data from different languages. We provide insights into what makes multilingual sequential learning particularly challenging. To surmount such challenges, we benchmark a rep- resentative set of cross-lingual continual learn- ing algorithms and analyze their knowledge preservation, accumulation, and generalization capabilities compared to baselines on carefully curated datastreams. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata, which go beyond conven- tional transfer learning. 1 Introduction With more than 7,000 languages spoken around the globe, downstream applications still lack proper linguistic resources across languages (Joshi et al., 2020), necessitating the use of transfer learning techniques that take advantage of data that is mis- matched to the application. In an effort to simplify architecture complexity and energy consumption, it is desirable to unify multi-lingual performance into a single, parameter- and memory-constrained model, and to allow this model to evolve, learning on multi-lingual training data as it becomes avail- able without having to pre-train or fine-tune from scratch. Such is the longstanding goal of language Figure 1: An overview of CCL: We use an example of a non-stationary datastream moving from high to low resource languages. Each bold and dashed box represents either a training or test data instance being fine-tuned or evaluated on, respectively. To support this problem setup, we evaluate the cross-lingual capabili- ties of continual approaches. Those capabilities include knowledge preservation on old languages, accumu- lation to the current language, and generalization to unseen languages at each point of the training. In ad- dition to that, we evaluate model utility at the end of continual learning. representation learning. Existing multi-lingual rep- resentations such as M-BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) are strong pil- lars in cross-lingual transfer learning, but if care is not taken when choosing how to fine-tune them, they can neglect to maximize transfer (Ruder et al., 2019) to new tasks or languages and are subject to forgetting (French, 1993), where performance decreases after exposure to new task or language. Most previous work that attempts to deal with the challenge …

Content truncated to 500 words