Bilingual Correspondence Recursive Autoencoder for Statistical Machine Translation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1248–1258, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Bilingual Correspondence Recursive Autoencoders for Statistical Machine Translation Jinsong Su1, Deyi Xiong2∗, Biao Zhang1, Yang Liu3, Junfeng Yao1, Min Zhang2 Xiamen University, Xiamen, P.R. China1 Soochow University, Suzhou, P.R. China2 Tsinghua University, Beijing, P.R. China3 {jssu, biaozhang, yao0010}@xmu.edu.cn {dyxiong, minzhang}@suda.edu.cn liuyang2011@tsinghua.edu.cn Abstract Learning semantic representations and tree structures of bilingual phrases is ben- eficial for statistical machine translation. In this paper, we propose a new neu- ral network model called Bilingual Corre- spondence Recursive Autoencoder (BCor- rRAE) to model bilingual phrases in trans- lation. We incorporate word alignments into BCorrRAE to allow it freely ac- cess bilingual constraints at different lev- els. BCorrRAE minimizes a joint objec- tive on the combination of a recursive au- toencoder reconstruction error, a structural alignment consistency error and a cross- lingual reconstruction error so as to not only generate alignment-consistent phrase structures, but also capture different lev- els of semantic relations within bilingual phrases. In order to examine the effective- ness of BCorrRAE, we incorporate both semantic and structural similarity features built on bilingual phrase representations and tree structures learned by BCorrRAE into a state-of-the-art SMT system. Exper- iments on NIST Chinese-English test sets show that our model achieves a substantial improvement of up to 1.55 BLEU points over the baseline. 1 Introduction Recently a variety of “deep architecture” ap- proaches, including autoencoders, have been suc- cessfully used in statistical machine translation (SMT) (Yang et al., 2013; Liu et al., 2013; Zou et al., 2013; Devlin et al., 2014; Tamura et al., 2014; Sundermeyer et al., 2014; Wang et al., 2014; Koˇcisk´y et al., 2014). Typically, these approaches represent words as dense, low-dimensional and ∗Corresponding author. real-valued vectors, i.e., word embeddings. How- ever, translation units in machine translation have long since shifted from words to phrases (se- quence of words), of which syntactic and se- mantic information cannot be adequately captured and represented by word embeddings. There- fore, learning compact vector representations for phrases or even longer expressions is more crucial for successful “deep” SMT. To address this issue, many efforts have been initiated on learning representations for bilingual phrases in the context of SMT, inspired by the suc- cess of work on monolingual phrase embeddings (Socher et al., 2010; Socher et al., 2011a; Socher et al., 2013b; Chen and Manning, 2014; Kalch- brenner et al., 2014; Kim, 2014). The learning process of bilingual phrase embeddings in these efforts is normally interacted and mingled with single or multiple essential components of SMT, e.g., with reordering models (Li et al., 2013), translation models (Cui et al., 2014; Zhang et al., 2014; Gao et al., 2014), or both language and translation models (Liu et al., 2014). In spite of their success, these approaches center around cap- turing relations between entire source and target phrases. They do not take into account internal phrase structures and bilingual correspondences of sub-phrases within source and target …

Content truncated to 500 words