A Teacher-Student Framework for Zero-Resource Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1925–1935 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1176 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1925–1935 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1176 A Teacher-Student Framework for Zero-Resource Neural Machine Translation Yun Chen†, Yang Liu‡∗, Yong Cheng+, Victor O.K. Li† †Department of Electrical and Electronic Engineering, The University of Hong Kong ‡State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China +Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China yun.chencreek@gmail.com; liuyang2011@tsinghua.edu.cn; chengyong3001@gmail.com; vli@eee.hku.hk Abstract While end-to-end neural machine transla- tion (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low-resource lan- guage pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on the assumption, our method is able to train a source-to-target NMT model (“stu- dent”) without parallel corpora available guided by an existing pivot-to-target NMT model (“teacher”) on a source-pivot par- allel corpus. Experimental results show that the proposed method significantly im- proves over a baseline pivot-based model by +3.0 BLEU points across various lan- guage pairs. 1 Introduction Neural machine translation (NMT) (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015), which directly models the translation process in an end-to-end way, has attracted intensive attention from the commu- nity. Although NMT has achieved state-of-the-art translation performance on resource-rich language pairs such as English-French and German-English (Luong et al., 2015; Jean et al., 2015; Wu et al., 2016; Johnson et al., 2016), it still suffers from the unavailability of large-scale parallel corpora for translating low-resource languages. Due to the large parameter space, neural models usually learn poorly from low-count events, resulting in a poor choice for low-resource language pairs. Zoph et ∗Corresponding author: Yang Liu. al. (2016) indicate that NMT obtains much worse translation quality than a statistical machine trans- lation (SMT) system on low-resource languages. As a result, a number of authors have endeav- ored to explore methods for translating language pairs without parallel corpora available. These methods can be roughly divided into two broad categories: multilingual and pivot-based. Firat et al. (2016b) present a multi-way, multilin- gual model with shared attention to achieve zero- resource translation. They fine-tune the attention part using pseudo bilingual sentences for the zero- resource language pair. Another direction is to develop a universal NMT model in multilingual scenarios (Johnson et al., 2016; Ha et al., 2016). They use parallel corpora of multiple languages to train one single model, which is then able to translate a language pair without parallel corpora available. Although these approaches prove to be effective, the combination of multiple …

Content truncated to 500 words