A Comparison between Count and Neural Network Models Based on Joint Translation and Reordering Sequences

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1401–1411, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Comparison between Count and Neural Network Models Based on Joint Translation and Reordering Sequences Andreas Guta, Tamer Alkhouli, Jan-Thorsten Peter, Joern Wuebker, Hermann Ney Human Language Technology and Pattern Recognition Group RWTH Aachen University Aachen, Germany {surname}@cs.rwth-aachen.de Abstract We propose a conversion of bilingual sentence pairs and the corresponding word alignments into novel linear se- quences. These are joint translation and reordering (JTR) uniquely defined sequences, combining interdepending lexical and alignment dependencies on the word level into a single framework. They are constructed in a simple manner while capturing multiple alignments and empty words. JTR sequences can be used to train a variety of models. We investigate the performances of n- gram models with modified Kneser-Ney smoothing, feed-forward and recur- rent neural network architectures when estimated on JTR sequences, and com- pare them to the operation sequence model (Durrani et al., 2013b). Evalua- tions on the IWSLT German→English, WMT German→English and BOLT Chinese→English tasks show that JTR models improve state-of-the-art phrase- based systems by up to 2.2 BLEU. 1 Introduction Standard phrase-based machine translation (Och et al., 1999; Zens et al., 2002; Koehn et al., 2003) uses relative frequencies of phrase pairs to esti- mate a translation model. The phrase table is ex- tracted from a bilingual text aligned on the word level, using e.g. GIZA++ (Och and Ney, 2003). Al- though the phrase pairs capture internal dependen- cies between the source and target phrases aligned to each other, they fail to model dependencies that extend beyond phrase boundaries. Phrase-based decoding involves concatenating target phrases. The burden of ensuring that the result is linguisti- cally consistent falls on the language model (LM). This work proposes word-based translation models that are potentially capable of capturing long-range dependencies. We do this in two steps: First, given bilingual sentence pairs and the asso- ciated word alignments, we convert the informa- tion into uniquely defined linear sequences. These sequenecs encode both word reordering and trans- lation information. Thus, they are referred to as joint translation and reordering (JTR) sequences. Second, we train an n-gram model with modi- fied Kneser-Ney smoothing (Chen and Goodman, 1998) on the resulting JTR sequences. This yields a model that fuses interdepending reordering and translation dependencies into a single framework. Although JTR n-gram models are closely re- lated to the operation sequence model (OSM) (Durrani et al., 2013b), there are three main dif- ferences. To begin with, the OSM employs min- imal translation units (MTUs), which are essen- tially atomic phrases. As the MTUs are extracted sentence-wise, a word can potentially appear in multiple MTUs. In order to avoid overlapping translation units, we define the JTR sequences on the level of words. Consequently, JTR se- quences have smaller vocabulary sizes than OSM sequences and lead to models with less sparsity. Moreover, we argue that JTR sequences offer a simpler reordering approach than operation se- quences, …

Content truncated to 500 words