Multilingual and cross-lingual document classification: A meta-learning approach

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1966–1976 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1966 Multilingual and cross-lingual document classification: A meta-learning approach Niels van der Heijden♣ Helen Yannakoudakis♠ Pushkar Mishra♦ Ekaterina Shutova♣ ♣ILLC, University of Amsterdam, the Netherlands ♠Dept. of Informatics, King’s College London, United Kingdom ♦Facebook AI, London, United Kingdom niels.vanderheijden@student.uva.nl, helen.yannakoudakis@kcl.ac.uk, pushkarmishra@fb.com, e.shutova@uva.nl Abstract The great majority of languages in the world are considered under-resourced for the suc- cessful application of deep learning methods. In this work, we propose a meta-learning ap- proach to document classification in limited- resource setting and demonstrate its effective- ness in two different settings: few-shot, cross- lingual adaptation to previously unseen lan- guages; and multilingual joint training when limited target-language data is available dur- ing training. We conduct a systematic compar- ison of several meta-learning methods, inves- tigate multiple settings in terms of data avail- ability and show that meta-learning thrives in settings with a heterogeneous task distribution. We propose a simple, yet effective adjustment to existing meta-learning methods which al- lows for better and more stable learning, and set a new state of the art on several languages while performing on-par on others, using only a small amount of labeled data. 1 Introduction There are more than 7000 languages around the world and, of them, around 6% account for 94% of the population.1 Even for the 6% most spoken languages, very few of them possess adequate re- sources for natural language research and, when they do, resources in different domains are highly imbalanced. Additionally, human language is dy- namic in nature: new words and domains emerge continuously and hence no model learned in a par- ticular time will remain valid forever. With the aim of extending the global reach of Natural Language Processing (NLP) technology, much recent research has focused on the devel- opment of multilingual models and methods to efficiently transfer knowledge across languages. 1https://www.ethnologue.com/statistics Among these advances are multilingual word vec- tors which aim to give word-translation pairs a sim- ilar encoding in some embedding space (Mikolov et al., 2013a; Lample et al., 2017). There has also been a lot of work on multilingual sentence and word encoders that either explicitly utilizes cor- pora of bi-texts (Artetxe and Schwenk, 2019; Lam- ple and Conneau, 2019) or jointly trains language models for many languages in one encoder (De- vlin et al., 2018; Conneau et al., 2019). Although great progress has been made in cross-lingual trans- fer learning, these methods either do not close the gap with performance in a single high-resource language (Artetxe and Schwenk, 2019; Conneau et al., 2019), e.g., because of cultural differences in languages which are not accounted for, or are impractically expensive (Lai et al., 2019). Meta-learning, or learning to learn (Schmidhu- ber, 1987; Bengio et al., 1990; Thrun and Pratt, 1998), is a learning paradigm which focuses on the quick adaption of a learner to new tasks. The idea is that by …

Content truncated to 500 words