A Deep Fusion Model for Domain Adaptation in Phrase-based MT

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3177–3187, Osaka, Japan, December 11-17 2016. A Deep Fusion Model for Domain Adaptation in Phrase-based MT Nadir Durrani ndurrani@qf.org.qa Shafiq Joty Ahmed Abdelali sjoty@qf.org.qa aabdelali@qf.org.qa Qatar Computing Research Institute – HBKU Hassan Sajjad hsajjad@qf.org.qa Abstract We present a novel fusion model for domain adaptation in Statistical Machine Translation. Our model is based on the joint source-target neural network (Devlin et al., 2014), and is learned by fusing in- and out-domain models. The adaptation is performed by backpropagating errors from the output layer to the word embedding layer of each model, subsequently adjusting pa- rameters of the composite model towards the in-domain data. On the standard tasks of translat- ing English-to-German and Arabic-to-English TED talks, we observed average improvements of +0.9 and +0.7 BLEU points, respectively over a competition grade phrase-based system. We also demonstrate improvements over existing adaptation methods. 1 Introduction The quality of machine translation systems is sensitive to the domain of the training data. More data, but out-of-domain data, may not be best suited and could even be harmful for specific translation tasks such as translating TED talks (Cettolo et al., 2014), patents (Fujii et al., 2010) and educational content (Guzm´an et al., 2013). This is because of the difference in stylistic variations, vocabulary choices and word sense ambiguities across genres. Domain adaptation aims at finding the optimal point, that maxi- mizes on the useful information available in the out-domain data, in favor of the in-domain data, while preventing it from degrading the performance of the system. This is either done by selecting a subset from the out-domain data, which is closer to the in-domain data (Matsoukas et al., 2009; Moore and Lewis, 2010), or by re-weighting the probability distribution in favor of the in-domain data (Foster and Kuhn, 2007; Sennrich, 2012). Recently, there has been a growing interest in deep neural networks (DNNs) and word embeddings with application to numerous NLP problems. The ability to generalize and to better capture non-local dependencies gives edge to the neural models over their traditional counter-part. Several researchers have also attempted to employ DNNs for domain adaptation in SMT. Duh et al. (2013) and Durrani et al. (2015) used DNNs for data selection. Joty et al. (2015) proposed a DNN-based adaptation model for SMT that regularizes the loss function with respect to the in-domain model. In this paper, we propose a deep fusion approach to domain adaptation for SMT. We use the Neural Network Joint Model (NNJM) proposed by Devlin et al. (2014) as our base model. However, rather than training the model on a plain concatenation of in- and out-domain data or a weighted concatenation (Joty et al., 2015), we first train in- and out-domain NNJM models, and then learn a composite model by readjusting their parameters through backpropagating errors from the output layer to the word embedding layer of each model. The intuition behind learning the models separately, is to learn in-domain model parameters without …

Content truncated to 500 words