Alignment-Based Neural Machine Translation

Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 54–65, Berlin, Germany, August 11-12, 2016. c⃝2016 Association for Computational Linguistics Alignment-Based Neural Machine Translation Tamer Alkhouli, Gabriel Bretschner, Jan-Thorsten Peter, Mohammed Hethnawi, Andreas Guta and Hermann Ney Human Language Technology and Pattern Recognition Group RWTH Aachen University, Aachen, Germany {surname}@cs.rwth-aachen.de Abstract Neural machine translation (NMT) has emerged recently as a promising statis- tical machine translation approach. In NMT, neural networks (NN) are directly used to produce translations, without re- lying on a pre-existing translation frame- work. In this work, we take a step to- wards bridging the gap between conven- tional word alignment models and NMT. We follow the hidden Markov model (HMM) approach that separates the align- ment and lexical models. We propose a neural alignment model and combine it with a lexical neural model in a log- linear framework. The models are used in a standalone word-based decoder that explicitly hypothesizes alignments during search. We demonstrate that our system outperforms attention-based NMT on two tasks: IWSLT 2013 German→English and BOLT Chinese→English. We also show promising results for re-aligning the train- ing data using neural models. 1 Introduction Neural networks have been gaining a lot of at- tention recently in areas like speech recognition, image recognition and natural language process- ing. In machine translation, NNs are applied in two main ways: In N-best rescoring, the neural model is used to score the first-pass decoding out- put, limiting the model to a fixed set of hypotheses (Le et al., 2012; Sundermeyer et al., 2014a; Hu et al., 2014; Guta et al., 2015). The second approach integrates the NN into decoding, potentially allow- ing it to directly determine the search space. There are two approaches to use neural mod- els in decoding. The first integrates the mod- els into phrase-based decoding, where the mod- els are used to score phrasal candidates hypothe- sized by the decoder (Vaswani et al., 2013; Devlin et al., 2014; Alkhouli et al., 2015). The second approach is referred to as neural machine trans- lation, where neural models are used to hypoth- esize translations, word by word, without relying on a pre-existing framework. In comparison to the former approach, NMT does not restrict NNs to predetermined translation candidates, and it does not depend on word alignment concepts that have been part of building state-of-the-art phrase-based systems. In such systems, the HMM and the IBM models developed more than two decades ago are used to produce Viterbi word alignments, which are used to build standard phrase-based systems. Existing NMT systems either disregard the no- tion of word alignments entirely (Sutskever et al., 2014), or rely on a probabilistic notion of align- ments (Bahdanau et al., 2015) independent of the conventional alignment models. Most recently, Cohn et al. (2016) designed neu- ral models that incorporate concepts like fertility and Markov conditioning into their structure. In this work, we also focus on the question whether conventional word alignment concepts can be used for NMT. In particular, …

Content truncated to 500 words