Recursive Autoencoders for ITG-Based Translation

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 567–577, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Recursive Autoencoders for ITG-based Translation Peng Li, Yang Liu and Maosong Sun State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China pengli09@gmail.com, {liuyang2011,sms}@tsinghua.edu.cn Abstract While inversion transduction grammar (ITG) is well suited for modeling ordering shifts between languages, how to make applying the two reordering rules (i.e., straight and inverted) dependent on actual blocks being merged remains a challenge. Unlike previous work that only uses boundary words, we pro- pose to use recursive autoencoders to make full use of the entire merging blocks alter- natively. The recursive autoencoders are ca- pable of generating vector space representa- tions for variable-sized phrases, which enable predicting orders to exploit syntactic and se- mantic information from a neural language modeling’s perspective. Experiments on the NIST 2008 dataset show that our system sig- nificantly improves over the MaxEnt classifier by 1.07 BLEU points. 1 Introduction Phrase-based models (Koehn et al., 2003; Och and Ney, 2004) have been widely used in practical ma- chine translation (MT) systems due to their effec- tiveness, simplicity, and applicability. First, as se- quences of consecutive words, phrases are capable of memorizing local word selection and reorder- ing, making them an effective mechanism for trans- lating idioms or translations with word insertions or omissions. Moreover, n-gram language models can be seamlessly integrated into phrase-based de- coders since partial translations grow left to right in decoding. Finally, phrase-based systems can be applicable to most domains and languages, espe- cially for resource-scarce languages without high- accuracy parsers. However, as phrase-based decoding casts transla- tion as a string concatenation problem and permits arbitrary permutations, it proves to be NP-complete (Knight, 1999). Therefore, phrase reordering mod- eling has attracted intensive attention in the past decade (e.g., Och et al., 2004; Tillman, 2004; Zens et al., 2004; Al-Onaizan and Papineni, 2006; Xiong et al., 2006; Koehn et al., 2007; Galley and Man- ning, 2008; Feng et al., 2010; Green et al., 2010; Bisazza and Federico, 2012; Cherry, 2013). Among them, reordering models based on inver- sion transduction grammar (ITG) (Wu, 1997) are one of the important ongoing research directions. As a formalism for bilingual modeling of sentence pairs, ITG is particularly well suited to predicting ordering shifts between languages. As a result, a number of authors have incorporated ITG into left- to-right decoding to constrain the reordering space and reported significant improvements (e.g., Zens et al., 2004; Feng et al., 2010). Along another line, Xiong et al. (2006) propose a maximum entropy (MaxEnt) reordering model based on ITG. They use the CKY algorithm to recursively merge two blocks (i.e., a pair of source and target strings) into larger blocks, either in a straight or an inverted order. Un- like lexicalized reordering models (Tillman, 2004; Koehn et al., 2007; Galley and Manning, 2008) that …

Content truncated to 500 words