Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2888–2901 August 1–6, 2021. ©2021 Association for Computational Linguistics 2888 Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning Cheonbok Park1,∗, Yunwon Tae2,∗,Taehee Kim3, Soyoung Yang3, Mohammad Azam Khan4, Lucy Park5,†, and Jaegul Choo3 1NAVER Corp,cbok.park@navercorp.com, 2Korea University, tyj204@korea.ac.kr 3KAIST, {taeheekim, sy yang, jchoo}@kaist.com 4DPDC, azam@dpdc.org.bd, 5Upstage AI Research, lucy@upstage.ai Abstract Unsupervised machine translation, which uti- lizes unpaired monolingual corpora as training data, has achieved comparable performance against supervised machine translation. How- ever, it still suffers from data-scarce domains. To address this issue, this paper presents a novel meta-learning algorithm for unsuper- vised neural machine translation (UNMT) that trains the model to adapt to another domain by utilizing only a small amount of training data. We assume that domain-general knowl- edge is a significant factor in handling data- scarce domains. Hence, we extend the meta- learning algorithm, which utilizes knowledge learned from high-resource domains, to boost the performance of low-resource UNMT. Our model surpasses a transfer learning-based ap- proach by up to 2-3 BLEU scores. Extensive experimental results show that our proposed algorithm is pertinent for fast adaptation and consistently outperforms other baselines. 1 Introduction Unsupervised neural machine translation (UNMT) leverages unpaired monolingual corpora for its training, without requiring an already labeled, parallel corpus. Recently, the state of the art in UNMT (Conneau and Lample, 2019; Song et al., 2019; Ren et al., 2019) has achieved comparable performances against supervised neural machine translation (NMT) approaches. In contrast to super- vised NMT, which uses a parallel corpus, training the UNMT model requires a significant number of monolingual sentences (e.g., 1M-3M sentences). However, the prerequisite limits UNMT’s appli- cability to low-resource domains, especially for ∗ equal contributions † This work done in NAVER Corp. Our code is avilable at https://github.com/ papago-lab/MetaGUMT domain-specific document translation tasks. Since gathering or creating those documents requires do- main specific knowledge, the monolingual data themselves are scarce and expensive. In addition, the minority languages (e.g., Uzbek and Nepali) make the problem of data scarcity even worse. Yet, UNMT for low-resource domains is not an actively explored field. One naive approach is to train a model on high-resource domains (e.g., econ- omy and sports) while hoping the model will gen- eralize on an unseen low-resource domain (e.g., medicine). However, recent studies have shown that non-trivial domain mismatch can significantly cause low translation accuracy on supervised NMT tasks (Koehn and Knowles, 2017). Another reasonable approach is transfer learning—particularly, domain adaptation—which has shown performance improvements in the su- pervised NMT literature (Freitag and Al-Onaizan, 2016; Zeng et al., 2019). In this approach, the model is first pretrained using data from existing domains and then finetuned on a new domain. However, this approach can suffer from overfitting and catastrophic forgetting due to a small amount of training data and a large domain gap. As an effective method for handling a small amount of training data, meta-learning has shown …

Content truncated to 500 words