Machine Translation With Weakly Paired Documents

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4375–4384, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4375 Machine Translation With Weakly Paired Documents Lijun Wu1, Jinhua Zhu2, Fei Gao3, Di He4, Tao Qin5, Jianhuang Lai1, Tie-Yan Liu5 1School of Data and Computer Science, Sun Yat-sen University; 2 University of Science and Technology of China; 3 Institute of Computing Technology, Chinese Academy of Sciences; 4 Peking University, 5 Microsoft Research Asia; 1{wulijun3@mail2, stsljh@mail}.sysu.edu.cn, 2teslazhu@mail.ustc.edu.cn, 3gaofei17b@ict.ac.cn, 4di he@pku.edu.cn, 5{taoqin, tyliu}@microsoft.com Abstract Neural machine translation, which achieves near human-level performance in some languages, strongly relies on the large amounts of parallel sentences, which hinders its applicability to low-resource language pairs. Recent works explore the possibility of unsupervised machine translation with monolingual data only, leading to much lower accuracy compared with the supervised one. Observing that weakly paired bilingual documents are much easier to collect than bilingual sentences, e.g., from Wikipedia, news websites or books, in this paper, we investigate training translation models with weakly paired bilingual documents. Our approach contains two components. 1) We provide a simple approach to mine implicitly bilingual sentence pairs from document pairs which can then be used as supervised training signals. 2) We leverage the topic consistency of two weakly paired documents and learn the sentence translation model by constrain- ing the word distribution-level alignments. We evaluate our method on weakly paired documents from Wikipedia on six tasks, the widely used WMT16 German↔English, WMT13 Spanish↔English and Romanian↔English translation tasks.WMT16 We obtain 24.1/30.3, 28.1/27.6 and 30.1/27.6 BLEU points separately, outperforming pre- vious results by more than 5 BLEU points in each direction and reducing the gap between unsupervised translation and supervised translation up to 50%. 1 Introduction Neural machine translation (NMT) is a great suc- cess of deep learning for natural language process- ing (Koehn et al., 2003; Cho et al., 2014; Sutskev- er et al., 2014; Bahdanau et al., 2015; Jean et al., 2015; Vaswani et al., 2017; Gehring et al., 2017). It has significantly outperformed statistical ma- chine translation and reached near human-level performance for several language pairs (Wu et al., 2016; Hassan et al., 2018). Such breakthrough- s heavily depend on the availability of large s- cale bilingual sentence pairs. Taking WMT14 English→French task as an example, NMT us- es 38 million parallel sentence pairs for training (Vaswani et al., 2017). As such bilingual sentence pairs are costly to collect, the success of NMT is not fully realized on the vast majority of language pairs, especially for low-resource languages. Re- cently, Artetxe et al. (2017); Lample et al. (2017) tackle this challenge by training NMT models us- ing only monolingual data, which achieves consid- erably good accuracy but still far away from that of the state-of-the-art supervised models. While it is costly to collect bilingual sentence pairs by human translation, we notice that there exists many weakly paired bilingual documents on the Web. For example, for the …

Content truncated to 500 words