Biasing Attention-Based Recurrent Neural Networks Using External Alignment Information

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 108–117 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Biasing Attention-Based Recurrent Neural Networks Using External Alignment Information Tamer Alkhouli and Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract This work explores extending attention- based neural models to include alignment information as input. We modify the at- tention component to have dependence on the current source position. The at- tention model is then used as a lexical model together with an additional align- ment model to generate translation. The attention model is trained using external alignment information, and it is applied in decoding by performing beam search over the lexical and alignment hypothe- ses. The alignment model is used to score these alignment candidates. We demon- strate that the attention layer is capable of using the alignment information to im- prove over the baseline attention model that uses no such alignments. Our experi- ments are performed on two tasks: WMT 2016 English→Romanian and WMT 2017 German→English. 1 Introduction Neural machine translation (NMT) has emerged recently as a successful end-to-end statistical ma- chine translation approach. The best performing NMT systems use an attention mechanism that fo- cuses the attention of the decoder on parts of the source sentence (Bahdanau et al., 2015). The at- tention component is computed as an intermedi- ate part of the model, and is trained jointly with the rest of the model. The approach is appeal- ing because (1) it is end-to-end, where the neural model is trained from scratch without assistance from other trained models, and (2) the attention component is trained jointly with the rest of the model, requiring no pre-computed alignments. In this work, we raise the question whether the attention component is self-sufficient to attend to the source side, and if it can still benefit from ex- plicit dependence on the alignment information. To this end, we modify the attention model to bias the attention layer towards the alignment informa- tion, and evaluate the model in a generative frame- work consisting of two steps: alignment prediction followed by lexical translation. Two decades ago, (Vogel et al., 1996) applied hidden Markov models to machine translation. The idea was based on introducing word align- ments as hidden variables, while using the first- order Markov assumption to simplify the depen- dencies of the alignment sequence. The approach decomposed the translation process using a lexi- cal model and an alignment model. These mod- els were simple tables enumerating all possible translation and alignment combinations. Nowa- days, HMM is used with IBM models to gener- ate word alignments, which are needed to train phrase-based systems. Alkhouli et al. (2016) and Wang et al. (2017) apply the hidden Markov model decomposition using feedforward lexical and alignment neural network models. In this work, we are interested in using more expressive models. Namely, we lever- age attention models as lexical models and use them …

Content truncated to 500 words