Statistical Machine Translation Features with Multitask Tensor Networks

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 31–41, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Statistical Machine Translation Features with Multitask Tensor Networks Hendra Setiawan, Zhongqiang Huang, Jacob Devlin†∗, Thomas Lamar, Rabih Zbib, Richard Schwartz and John Makhoul Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138, USA †Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA {hsetiawa,zhuang,tlamar,rzbib,schwartz,makhoul}@bbn.com jdevlin@microsoft.com Abstract We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non- local translation phenomena. Second, we augment the architecture of the neural net- work with tensor layers that capture im- portant higher-order interaction among the network units. Third, we apply multitask learning to estimate the neural network parameters jointly. Each of our proposed methods results in significant improve- ments that are complementary. The over- all improvement is +2.7 and +1.8 BLEU points for Arabic-English and Chinese- English translation over a state-of-the-art system that already includes neural net- work features. 1 Introduction Recent advances in applying Neural Networks to Statistical Machine Translation (SMT) have gen- erally taken one of two approaches. They ei- ther develop neural network-based features that are used to score hypotheses generated from tra- ditional translation grammars (Sundermeyer et al., 2014; Devlin et al., 2014; Auli et al., 2013; Le et al., 2012; Schwenk, 2012), or they implement the whole translation process as a single neu- ral network (Bahdanau et al., 2014; Sutskever et al., 2014). The latter approach, sometimes re- ferred to as Neural Machine Translation, attempts to overhaul SMT, while the former capitalizes on the strength of the current SMT paradigm and leverages the modeling power of neural networks to improve the scoring of hypotheses generated ∗* Research conducted when the author was at BBN. by phrase-based or hierarchical translation rules. This paper adopts the former approach, as n-best scores from state-of-the-art SMT systems often suggest that these systems can still be significantly improved with better features. We build on (Devlin et al., 2014) who proposed a simple yet powerful feedforward neural network model that estimates the translation probability conditioned on the target history and a large win- dow of source word context. We take advantage of neural networks’ ability to handle sparsity, and to infer useful abstract representations automati- cally. At the same time, we address the challenge of learning the large set of neural network param- eters. In particular, • We develop new Neural Network Features to model non-local translation phenomena related to word reordering. Large fully- lexicalized contexts are used to model these phenomena effectively, making the use of neural networks essential. All of the features are useful individually, and their combination results in significant improvements (Section 2). • We use a Tensor Neural Network Architecture (Yu et al., 2012) to automatically learn com- plex pairwise interactions between the …

Content truncated to 500 words