Using joint models or domain adaptation in statistical machine translation

Using Joint Models for Domain Adaptation in Statistical Machine Translation Nadir Durrani Hassan Sajjad Shafiq Joty Ahmed Abdelali Stephan Vogel Qatar Computing Research Institute – Hamad Bin Khalifa University {ndurrani,hsajjad,sjoty,aabdelali,svogel}@qf.org.qa Abstract Joint models have recently shown to improve the state-of-the-art in machine translation (MT). We apply EM-based mixture modeling and data selection techniques using two joint models, namely the Operation Sequence Model or OSM — an ngram-based translation and reordering model, and the Neural Network Joint Model or NNJM — a continuous space translation model, to carry out domain adaptation for MT. The diversity of the two models, OSM with inherit reordering information and NNJM with continuous space modeling makes them interesting to be explored for this task. Our contribution in this paper is fusing the existing known techniques (linear interpolation, cross-entropy) with the state-of-the-art MT models (OSM, NNJM). On a standard task of translating German-to-English and Arabic-to-English IWSLT TED talks, we observed statistically significant improvements of up to +0.9 BLEU points. 1 Introduction Parallel data required to train Statistical Machine Translation (SMT) systems is often inade- quate, and is typically collected opportunistically from wherever it is available. The conven- tional wisdom is that more data improves the translation quality. Additional data however, may not be best suited for tasks such as translating TED talks (Cettolo et al., 2014) or patents (Fujii et al., 2010) or educational content (Abdelali et al., 2014), and often come with the challenges of dealing with word-sense ambiguities and stylistic variance of other domains. When additional data, later referred as out-domain, is much larger than in-domain, the resultant distribution can get biased towards out-domain, yielding a sub-optimal system. Domain adaptation aims to pre- serve the identity of the in-domain data while using the best of the out-domain data. This is done by selecting a subset from the out-domain data, which is closer to the in-domain (Mat- soukas et al., 2009; Moore and Lewis, 2010), or by re-weighting the probability distribution in favor of the in-domain data (Foster and Kuhn, 2007; Sennrich, 2012). Bilingual sequence models (Mari˜no et al., 2006) have shown to be effective in improv- ing the quality of machine translation and have achieved state-of-the-art performance recently (Le et al., 2012; Durrani et al., 2013; Devlin et al., 2014). Their ability to capture non-local dependencies makes them superior to the traditional phrase-based models, which do not con- sider contextual information across phrasal boundaries. Two such models that we explore in this paper are (i) the Operation Sequence Model or OSM (Durrani et al., 2011) — a markov translation model that integrates reordering, and (ii) the Neural Network Joint Model or NNJM (Devlin et al., 2014) — a continuous space model that learns neural network over augmented streams of source and target sequences. Both models are used as additional language model (LM) features inside the SMT decoder. Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 117 The diversity of the two models, i.e., OSM with …

Content truncated to 500 words