Evaluating morphological typology in zero-shot cross-lingual transfer

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3136–3153 August 1–6, 2021. ©2021 Association for Computational Linguistics 3136 Evaluating morphological typology in zero-shot cross-lingual transfer Antonio Mart´ınez-Garc´ıa Universitat de Barcelona ao.martinez.garcia @gmail.com Toni Badia Universitat Pompeu Fabra tbadia@upf.edu Jeremy Barnes University of Oslo jeremycb@ifi.uio.no Abstract Cross-lingual transfer has improved greatly through multi-lingual language model pretrain- ing, reducing the need for parallel data and increasing absolute performance. However, this progress has also brought to light the differences in performance across languages. Specifically, certain language families and ty- pologies seem to consistently perform worse in these models. In this paper, we address what effects morphological typology has on zero-shot cross-lingual transfer for two tasks: Part-of-speech tagging and sentiment analysis. We perform experiments on 19 languages from four language typologies (fusional, isolating, agglutinative, and introflexive) and find that transfer to another morphological type gener- ally implies a higher loss than transfer to an- other language with the same morphological typology. Furthermore, POS tagging is more sensitive to morphological typology than sen- timent analysis and, on this task, models per- form much better on fusional languages than on the other typologies. 1 Introduction Cross-lingual transfer uses available annotated re- sources in a source language to learn a model that will transfer to a target language. Earlier work used machine translation (Mihalcea et al., 2007), paral- lel data (Pad´o and Lapata, 2009), or delexicalized models (Zeman and Resnik, 2008; McDonald et al., 2011; Søgaard, 2011) to bridge the gap between languages. However, recent improvements (Devlin et al., 2019) have reduced the need for parallel data, instead relying on multi-lingual language models, trained on the concatenation of monolingual cor- pora. Fine-tuning these multilingual language mod- els on a task in a source language can lead to strong performance when applied directly to the target- language task (zero-shot transfer). This progress has uncovered gaps in perfor- mance, as transfer is generally easier between simi- lar languages, and some language families consis- tently perform worse (Artetxe et al., 2020; Conneau et al., 2020a). So far, however, the analysis of these differences has only been anecdotal, rather than centered as a research question of its own merit. For these cases, linguistic typology has important implications, as it gives us ways to quantify the sim- ilarity of languages along certain variables, such as shared morphological or syntactic features (Bender, 2013). While previous work has studied the effects of morphological typology on language modeling (Gerz et al., 2018; Cotterell et al., 2018; Mielke et al., 2019), this effect on cross-lingual transfer has not been looked at in detail. In this paper we attempt to answer (RQ1) to what degree morphological typology affects the performance of state-of-the-art cross-lingual mod- els, (RQ2) whether morphological typology has a stronger effect than other variables, e.g., the amount of data for pretraining the LM or domain mis- matches between source and target, (RQ3) whether there is a different effect on a …

Content truncated to 500 words