Neural Network-based Word Alignment through Score Aggregation

Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 66–73, Berlin, Germany, August 11-12, 2016. c⃝2016 Association for Computational Linguistics Neural Network-based Word Alignment through Score Aggregation Jo¨el Legrand1,2,† and Michael Auli3 and Ronan Collobert3 1 Idiap Research Institute, Martigny, Switzerland 2 Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Lausanne, Switzerland 3 Facebook AI Research, Menlo Park Abstract We present a simple neural network for word alignment that builds source and tar- get word window representations to com- pute alignment scores for sentence pairs. To enable unsupervised training, we use an aggregation operation that summarizes the alignment scores for a given target word. A soft-margin objective increases scores for true target words while de- creasing scores for target words that are not present. Compared to the popular Fast Align model, our approach improves alignment accuracy by 7 AER on English- Czech, by 6 AER on Romanian-English and by 1.7 AER on English-French align- ment. 1 Introduction Word alignment is the task of finding the cor- respondence between source and target words in a pair of sentences that are translations of each other. Generative models for this task (Brown et al., 1990; Och and Ney, 2003; Vogel et al., 1996) still form the basis for many machine translation systems (Koehn et al., 2003; Chiang, 2007). Recent neural approaches include Yang et al. (2013) who introduce a feed-forward network- based model trained on alignments that were gen- erated by a traditional generative model. This treats potentially erroneous alignments as super- vision. Tamura et al. (2014) sidesteps this issue by negative sampling to train a recurrent-neural net- work on unlabeled data. They optimize a global loss that requires an expensive beam search to ap- proximate the sum over all alignments. †This work was conducted while the first author did an internship at Facebook AI Research. In this paper we introduce a word alignment model that is simpler in structure and which re- lies on a more tractable training procedure. Our model is a neural network that extracts context in- formation from source and target sentences and then computes simple dot products to estimate alignment links. Our objective function is word- factored and does not require the expensive com- putation associated with global loss functions. The model can be easily trained on unlabeled data via a novel but simple aggregation operation which has been successfully applied in the computer vision literature (Pinheiro and Collobert, 2015). The ag- gregation combines the scores of all source words for a particular target word and promotes source words which are likely to be aligned with a given target word according to the knowledge the model has learned so far. At test time, the aggregation op- eration is removed and source words are aligned to target words by choosing the highest scoring can- didates (§2, §3). We evaluate several forms for our aggregation operation such as computing the sum, max and LogSumExp over alignment scores. Results on English-French, English-Romanian, and Czech- English alignment show that our …

Content truncated to 500 words