Learning New Semi-Supervised Deep Auto-encoder Features for Statistical Machine Translation

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 122–132, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Learning New Semi-Supervised Deep Auto-encoder Features for Statistical Machine Translation Shixiang Lu, Zhenbiao Chen, Bo Xu Interactive Digital Media Technology Research Center (IDMTech) Institute of Automation, Chinese Academy of Sciences, Beijing, China {shixiang.lu,zhenbiao.chen,xubo}@ia.ac.cn Abstract In this paper, instead of designing new fea- tures based on intuition, linguistic knowl- edge and domain, we learn some new and effective features using the deep auto- encoder (DAE) paradigm for phrase-based translation model. Using the unsupervised pre-trained deep belief net (DBN) to ini- tialize DAE’s parameters and using the in- put original phrase features as a teacher for semi-supervised fine-tuning, we learn new semi-supervised DAE features, which are more effective and stable than the unsuper- vised DBN features. Moreover, to learn high dimensional feature representation, we introduce a natural horizontal compo- sition of more DAEs for large hidden lay- ers feature learning. On two Chinese- English tasks, our semi-supervised DAE features obtain statistically significant im- provements of 1.34/2.45 (IWSLT) and 0.82/1.52 (NIST) BLEU points over the unsupervised DBN features and the base- line features, respectively. 1 Introduction Recently, many new features have been explored for SMT and significant performance have been obtained in terms of translation quality, such as syntactic features, sparse features, and reordering features. However, most of these features are man- ually designed on linguistic phenomena that are related to bilingual language pairs, thus they are very difficult to devise and estimate. Instead of designing new features based on in- tuition, linguistic knowledge and domain, for the first time, Maskey and Zhou (2012) explored the possibility of inducing new features in an unsuper- vised fashion using deep belief net (DBN) (Hinton et al., 2006) for hierarchical phrase-based trans- lation model. Using the 4 original phrase fea- tures in the phrase table as the input features, they pre-trained the DBN by contrastive divergence (Hinton, 2002), and generated new unsupervised DBN features using forward computation. These new features are appended as extra features to the phrase table for the translation decoder. However, the above approach has two major shortcomings. First, the input original features for the DBN feature learning are too simple, the limited 4 phrase features of each phrase pair, such as bidirectional phrase translation probabil- ity and bidirectional lexical weighting (Koehn et al., 2003), which are a bottleneck for learning ef- fective feature representation. Second, it only uses the unsupervised layer-wise pre-training of DBN built with stacked sets of Restricted Boltzmann Machines (RBM) (Hinton, 2002), does not have a training objective, so its performance relies on the empirical parameters. Thus, this approach is un- stable and the improvement is limited. In this pa- per, we strive to effectively address the above two shortcomings, and systematically explore the pos- sibility of learning new features using deep (multi- layer) neural networks (DNN, which is usually re- ferred under the name Deep Learning) for SMT. To address the first shortcoming, …

Content truncated to 500 words