Exploring Cross-Lingual Transfer Learning with Unsupervised Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2011–2020 August 1–6, 2021. ©2021 Association for Computational Linguistics 2011 Exploring Cross-Lingual Transfer Learning with Unsupervised Machine Translation Chao Wang ∗ York University Toronto, Canada chwang@eecs.yorku.ca Judith Gaspers Amazon Alexa AI Aachen, Germany gaspers@amazon.com Quynh Do Amazon Alexa AI Aachen, Germany doquynh@amazon.com Hui Jiang York University Toronto, Canada hj@eecs.yorku.ca Abstract In Natural Language Understanding (NLU), to facilitate Cross-Lingual Transfer Learning (CLTL), especially CLTL between distant lan- guages, we integrate CLTL with Machine Translation (MT), and thereby propose a novel CLTL model named Translation Aided Lan- guage Learner (TALL). TALL is constructed as a standard transformer, where the encoder is a pre-trained multilingual language model. The training of TALL includes an MT-oriented pre-training and an NLU-oriented fine-tuning. To make use of unannotated data, we imple- ment the recently proposed Unsupervised Ma- chine Translation (UMT) technique in the MT- oriented pre-training of TALL. The experimen- tal results show that the application of UMT enables TALL to consistently achieve better CLTL performance than our baseline model, which is the pre-trained multilingual language model serving as the encoder of TALL, with- out using more annotated data, and the perfor- mance gain is relatively prominent in the case of distant languages. 1 Introduction Virtual assistants, such as Amazon Alexa, Apple Siri, and Google Assistant, are increasingly popu- lar due to the convenience they bring to customers. A core function of virtual assistants is Natural Lan- guage Understanding (NLU), which is a combo of slot filling and intent classification. NLU models behind virtual assistants are generally trained in a supervised manner, which requires a large amount of annotated data. Collecting annotated data is not a big deal for high-resource languages, but difficult ∗Work done during internship at Amazon Alexa AI. or even impossible for low-resource languages. As a result, when ported to a low-resource language, an NLU model may suffer from the so-called “data hungriness” (van der Ploeg et al., 2014). This prob- lem can be alleviated by conducting Cross-Lingual Transfer Learning (CLTL) (Yarowsky et al., 2001), where annotated data in a high-resource source lan- guage is used to bootstrap an NLU model aimed at a low-resource target language. The key to CLTL is to learn a shared representation space for the given source-target language pair. A traditional way to achieve this goal is to leverage cross-lingual word embeddings, which are obtained by mapping the words in both languages to a shared word embedding space (Zhang et al., 2017; Con- neau et al., 2017; Artetxe et al., 2018a; Chen et al., 2018; Chen and Cardie, 2018; Chen et al., 2019). However, most studies on this topic only consider similar languages (e.g. English-German) but ignore distant languages (e.g. English-Japanese), since it is more challenging to conduct CLTL between dis- tant languages than between similar languages. Re- cently, contextualized word embeddings generated by pre-trained language models have shown signif- icant advantages over ordinary word embeddings (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). For …

Content truncated to 500 words