Bilingual Methods for Adaptive Training Data Selection for Machine Translation

Bilingual Methods for Adaptive Training Data Selection for Machine Translation Boxing Chen Boxing.Chen@nrc-cnrc.gc.ca Roland Kuhn Roland.Kuhn@nrc-cnrc.gc.ca George Foster George.Foster@nrc-cnrc.gc.ca Colin Cherry Colin.Cherry@nrc-cnrc.gc.ca National Research Council Canada, Ottawa, ON, Canada Fei Huang feihuang@fb.com Facebook, New York, NY, USA Abstract In this paper, we propose a new data selection method which uses semi-supervised convolu- tional neural networks based on bitokens (Bi-SSCNNs) for training machine translation sys- tems from a large bilingual corpus. In earlier work, we devised a data selection method based on semi-supervised convolutional neural networks (SSCNNs). The new method, Bi-SSCNN, is based on bitokens, which use bilingual information. When the new methods are tested on two translation tasks (Chinese-to-English and Arabic-to-English), they significantly outper- form the other three data selection methods in the experiments. We also show that the Bi- SSCNN method is much more effective than other methods in preventing noisy sentence pairs from being chosen for training. More interestingly, this method only needs a tiny amount of in-domain data to train the selection model, which makes fine-grained topic-dependent transla- tion adaptation possible. In the follow-up experiments, we find that neural machine translation (NMT) is more sensitive to noisy data than statistical machine translation (SMT). Therefore, Bi-SSCNN which can effectively screen out noisy sentence pairs, can benefit NMT much more than SMT.We observed a BLEU improvement over 3 points on an English-to-French WMT task when Bi-SSCNNs were used. 1 Introduction When building a statistical machine translation (SMT) system, it is important to choose bilin- gual training data that are of high quality 1 and that are typical of the domain in which the SMT system will operate. In previous work, these two goals of data selection, i.e., picking high-quality data and picking data that ensure the SMT system is well-adapted to a given do- main, have often been achieved separately. For instance, the papers (Munteanu and Marcu, 2005; Khadivi and Ney, 2005; Okita et al., 2009; Jiang et al., 2010; Denkowski et al., 2012) focus on reducing the noise in the data. They use different scoring functions, such as language model perplexity, word alignment score, or IBM model 1 score, to score each sentence pair, top scored sentence pairs are selected. While the papers (Zhao et al., 2004; L¨u et al., 2007; Yasuda et al., 2008; Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Axelrod et al., 2015) focus on domain adaptation. They all select monolingual or bilingual data that 1Within each sentence pair, the target-language sentence is a good translation of the source sentence ������������������������������������������������������ ���������������������������������������� are similar to the in-domain data according to some criterion. These state-of-the-art adaptive data selection approaches (Axelrod et al., 2011; Duh et al., 2013; Axelrod et al., 2015) search for bilingual parallel sentences using the difference in language model perplexity between two language models trained on in-domain and out-domain data, respectively. Furthermore, (Duh et al., 2013) extends these approaches from n-gram models to recurrent neural network lan- guage models (Mikolov et al., 2010). While some previous work considers …

Content truncated to 500 words