Multilingual Continual Learning Approaches for Text Classification

Proceedings of Recent Advances in Natural Language Processing, pages 864–870 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_093 864 Multilingual Continual Learning Approaches for Text Classification Karan Praharaj RevealIrina Matveeva Chicago, IL {kpraharaj,imatveeva}@revealdata.com Abstract Multilingual continual learning is important for models that are designed to be deployed over long periods of time and are required to be up- dated when new data becomes available. Such models are continually applied to new unseen data that can be in any of the supported lan- guages. One challenge in this scenario is to ensure consistent performance of the model throughout the deployment lifecycle, beginning from the moment of first deployment. We em- pirically assess the strengths and shortcomings of some continual learning methods in a multi- lingual setting across two tasks. 1 Introduction There is a substantial amount of research in con- tinual learning that studies how large language models can be trained in multiple steps. Task- incremental learning (Kirkpatrick et al., 2017; Chaudhry et al., 2019; Lopez-Paz and Ranzato, 2017), class-incremental learning (Wu et al., 2019), domain-incremental learning (Wang et al., 2022) and language-incremental learning (Badola et al., 2022; Castellucci et al., 2021; M’hamdi et al., 2022; Praharaj and Matveeva, 2022) have been exten- sively studied (van de Ven and Tolias, 2019) in the realm of continual learning. The motivation is for the same model to be trained on various tasks/classes/domains/languages not only to avoid having individual models for each task, class, do- main, or language but also to improve the overall model performance. One of the challenges in continual learning is to counteract the tendency of large language models to “forget” previously learned information when trained on new data. In multilingual models, the performance of the model on languages fine-tuned in the past tends to decrease or can even result in catastrophic forgetting (Mccloskey and Cohen, 1989; French, 1999). Task-incremental learning, class-incremental learning, and domain-incremental learning re- search typically focus on few-step continual learn- ing, where the model is incrementally fine-tuned with a few tasks, classes, or domains. while language-incremental learning or multilingual con- tinual learning offers an opportunity to do continual learning over dozens of steps. We will use the term “multilingual continual learning” in this paper. This work addresses the performance of training multilingual models over many fine-tuning steps. We analyze the main existing approaches, provide a summary of their performance on two multilingual datasets and provide an analysis of the trade-offs for selecting one approach over another. For this work, we develop a test scenario that is representative of the practical setting: we update multilingual classification models over a large num- ber of steps with small amounts of training data, and the training data is in different languages. We consider three dimensions in which the ap- proaches differ: the amount of previously seen data that is used in fine-tuning steps, the training time, and the fine-tuning of data in different languages. The main contributions are as follows: • We study different multilingual continual learning approaches on two datasets with a long …

Content truncated to 500 words