Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

Proceedings of NAACL-HLT 2016, pages 876–885, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Incorporating Structural Alignment Biases into an Attentional Neural Translation Model Trevor Cohn and Cong Duy Vu Hoang and Ekaterina Vymolova University of Melbourne Melbourne, VIC, Australia tcohn@unimelb.edu.au and {vhoang2,evylomova}@student.unimelb.edu.au Kaisheng Yao Microsoft Research Redmond, WA, USA kaisheng.YAO@microsoft.com Chris Dyer Carnegie Mellon University Pittsburgh, PA, USA cdyer@cmu.edu Gholamreza Haffari Monash University Clayton, VIC, Australia gholamreza.haffari@monash.edu Abstract Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. How- ever their modelling formulation is overly simplistic, and omits several key inductive bi- ases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agree- ment over translation directions. We show im- provements over a baseline attentional model and standard phrase-based model over sev- eral language pairs, evaluating on difficult lan- guages in a low resource setting. 1 Introduction Recently, models of end-to-end machine translation based on neural network classification have been shown to produce excellent translations, rivalling or in some cases surpassing traditional statistical ma- chine translation systems (Kalchbrenner and Blun- som, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). This is despite the neural approaches using an overall simpler model, with fewer assumptions about the learning and prediction problem. Broadly, neural approaches are based around the notion of an encoder-decoder (Sutskever et al., 2014), in which the source language is encoded into a distributed representation, followed by a decoding step which generates the target translation. We focus on the attentional model of translation (Bahdanau et al., 2015) which uses a dynamic representation of the source sentence while allowing the decoder to attend to different parts of the source as it gener- ates the target sentence. The attentional model raises intriguing opportunities, given the correspondence between the notions of attention and alignment in traditional word-based machine translation models (Brown et al., 1993). In this paper we map modelling biases from word based translation models into the attentional model, such that known linguistic elements of translation can be better captured. We incorporate absolute po- sitional bias whereby word order tends to be simi- lar between the source sentence and its translation (e.g., IBM Model 2 and (Dyer et al., 2013)), fer- tility whereby each instance of a source word type tends to be translated into a consistent number of target tokens (e.g., IBM Models 3, 4, 5), relative position bias whereby prior preferences for mono- tonic alignments/attention can be encouraged (e.g., IBM Model 4, 5 and HMM-based Alignment (Vogel et al., 1996)), and alignment consistency whereby the attention in both translation directions are en- couraged to agree (e.g. symmetrisation heuristics (Och and Ney, 2003) or joint modelling (Liang et al., 2006; Ganchev et al., 2008)). We provide an empirical analysis of incorporat- ing the above structural biases into the attentional model, considering low resource translation sce- nario over four language-pairs. Our results …

Content truncated to 500 words