Additive Neural Networks for Statistical Machine Translation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 791–801, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Additive Neural Networks for Statistical Machine Translation Lemao Liu1, Taro Watanabe2, Eiichiro Sumita2, Tiejun Zhao1 1School of Computer Science and Technology Harbin Institute of Technology (HIT), Harbin, China 2National Institute of Information and Communication Technology (NICT) 3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, Japan {lmliu | tjzhao}@mtlab.hit.edu.cn {taro.watanabe | eiichiro.sumita}@nict.go.jp Abstract Most statistical machine translation (SMT) systems are modeled using a log- linear framework. Although the log-linear model achieves success in SMT, it still suffers from some limitations: (1) the features are required to be linear with respect to the model itself; (2) features cannot be further interpreted to reach their potential. A neural network is a reasonable method to address these pitfalls. However, modeling SMT with a neural network is not trivial, especially when taking the decoding efficiency into consideration. In this paper, we propose a variant of a neural network, i.e. additive neural networks, for SMT to go beyond the log-linear translation model. In addition, word embedding is employed as the input to the neural network, which encodes each word as a feature vector. Our model outperforms the log-linear translation models with/without embed- ding features on Chinese-to-English and Japanese-to-English translation tasks. 1 Introduction Recently, great progress has been achieved in SMT, especially since Och and Ney (2002) pro- posed the log-linear model: almost all the state- of-the-art SMT systems are based on the log-linear model. Its most important advantage is that arbi- trary features can be added to the model. Thus, it casts complex translation between a pair of lan- guages as feature engineering, which facilitates re- search and development for SMT. Regardless of how successful the log-linear model is in SMT, it still has some shortcomings. This joint work was done while the first author visited NICT. On the one hand, features are required to be lin- ear with respect to the objective of the translation model (Nguyen et al., 2007), but it is not guaran- teed that the potential features be linear with the model. This induces modeling inadequacy (Duh and Kirchhoff, 2008), in which the translation per- formance may not improve, or may even decrease, after one integrates additional features into the model. On the other hand, it cannot deeply in- terpret its surface features, and thus can not ef- ficiently develop the potential of these features. What may happen is that a feature p does initially not improve the translation performance, but after a nonlinear operation, e.g. log(p), it does. The reason is not because this feature is useless but the model does not efficiently interpret and represent it. Situations such as this confuse explanations for feature designing, since it is unclear whether such a feature contributes to a translation or not. A neural network (Bishop, 1995) is a reason- able method to overcome the above shortcomings. However, it should take constraints, e.g. the de- coding efficiency, into account in SMT. Decod- …

Content truncated to 500 words