DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 385–406 May 2-6, 2023 ©2023 Association for Computational Linguistics DiTTO : A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer Shanu Kumar Abbaraju Soujanya Sandipan Dandapat Sunayana Sitaram Monojit Choudhury Microsoft Corporation, India {shankum,asoujanya,sadandap,susitara,monojitc}@microsoft.com Abstract Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low- resource languages. In this work, we envision languages as domains for improving zero-shot transfer by jointly reducing the feature incon- gruity between the source and the target lan- guage and increasing the generalization capa- bilities of pre-trained multilingual transformers. We show that our approach, DiTTO , signifi- cantly outperforms the standard zero-shot fine- tuning method on multiple datasets across all languages using solely unlabeled instances in the target language. Empirical results show that jointly reducing feature incongruity for multi- ple target languages is vital for successful cross- lingual transfer. Moreover, our model enables better cross-lingual transfer than standard fine- tuning methods, even in the few-shot setting. 1 Introduction Due to the emergence of pre-trained Mas- sively Multilingual Transformers (MMTs) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2020), zero-shot cross-lingual transfer (Hu et al., 2020; Ruder et al., 2021; Lauscher et al., 2020; Ansell et al., 2021; Pfeiffer et al., 2022) has received significant atten- tion in the NLP community. This approach orig- inated due to the skew in resource distribution in languages (Joshi et al., 2020), with most languages of the world having a scarcity of labeled data. Zero- shot transfer involves fine-tuning the MMT with task-specific data in one or more source languages, followed by evaluation on target languages whose labeled instances are not used during fine-tuning. Accurate zero-shot transfer is crucially important for MMTs to be useful for low-resource languages. The performance of MMTs drops in the fol- lowing two cases - when the source and target languages exhibit dissimilar typological features, or when the size of pre-training data in the tar- get language is limited (Lauscher et al., 2020; Ebrahimi et al., 2022). Two common techniques to improve zero-shot performance include few-shot cross-lingual transfer (Lauscher et al., 2020; Kumar et al., 2022) and the translate-train approach (Ruder et al., 2021; Ahuja et al., 2022). Several studies have been conducted comparing these approaches, of which (Ahuja et al., 2022) concludes that if the cost of machine translation is greater than zero, the optimal and lowest-cost performance is achieved with at least some manually labeled data (i.e. the few-shot method). Since annotating data is expen- sive for many languages (Dandapat et al., 2009; Sabou et al., 2012; Fort, 2016), we investigate im- proving cross-lingual zero-shot transfer using only unlabelled data in this paper. Zero-shot Cross-lingual Transfer has been iden- tified as an under-specified optimization problem (Wu et al., 2022). A majority of the solutions re- ports a high performance on the source language but fluctuating performance on target languages. Wu …

Content truncated to 500 words