Non-linear Learning for Statistical Machine Translation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 825–835, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Non-linear Learning for Statistical Machine Translation Shujian Huang, Huadong Chen, Xinyu Dai and Jiajun Chen State Key Laboratory for Novel Software Technology Nanjing University Nanjing 210023, China {huangsj, chenhd, daixy, chenjj}@nlp.nju.edu.cn Abstract Modern statistical machine translation (SMT) systems usually use a linear com- bination of features to model the quality of each translation hypothesis. The linear combination assumes that all the features are in a linear relationship and constrains that each feature interacts with the rest fea- tures in an linear manner, which might limit the expressive power of the model and lead to a under-fit model on the cur- rent data. In this paper, we propose a non- linear modeling for the quality of transla- tion hypotheses based on neural networks, which allows more complex interaction between features. A learning framework is presented for training the non-linear mod- els. We also discuss possible heuristics in designing the network structure which may improve the non-linear learning per- formance. Experimental results show that with the basic features of a hierarchical phrase-based machine translation system, our method produce translations that are better than a linear model. 1 Introduction One of the core problems in the research of statis- tical machine translation is the modeling of trans- lation hypotheses. Each modeling method defines a score of a target sentence e = e1e2...ei...eI, given a source sentence f = f1f2...fj...fJ, where each ei is the ith target word and fj is the jth source word. The well-known modeling method starts from the Source-Channel model (Brown et al., 1993)(Equation 1). The scoring of e decom- poses to the calculation of a translation model and a language model. Pr(e|f) = Pr(e)Pr(f|e)/Pr(f) (1) The modeling method is extended to log-linear models by Och and Ney (2002), as shown in Equa- tion 2, where hm(e|f) is the mth feature function and λm is the corresponding weight. Pr(e|f) = pλM 1 (e|f) = exp[∑M m=1 λmhm(e|f)] ∑ e′ exp[∑M m=1 λmhm(e′|f)] (2) Because the normalization term in Equation 2 is the same for all translation hypotheses of the same source sentence, the score of each hypothesis, de- noted by sL, is actually a linear combination of all features, as shown in Equation 3. sL(e) = M ∑ m=1 λmhm(e|f) (3) The log-linear models are flexible to incorpo- rate new features and show significant advantage over the traditional source-channel models, thus become the state-of-the-art modeling method and are applied in various translation settings (Yamada and Knight, 2001; Koehn et al., 2003; Chiang, 2005; Liu et al., 2006). It is worth noticing that log-linear models try to separate good and bad translation hypotheses us- ing a linear hyper-plane. However, complex inter- actions between features make it difficult to lin- early separate good translation hypotheses from bad ones (Clark et al., 2014). Taking common features in a typical …

Content truncated to 500 words