Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1370–1380, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Fast and Robust Neural Network Joint Models for Statistical Machine Translation Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138, USA {jdevlin,rzbib,zhuang,tlamar,schwartz,makhoul}@bbn.com Abstract Recent work has shown success in us- ing neural network language models (NNLMs) as features in MT systems. Here, we present a novel formulation for a neural network joint model (NNJM), which augments the NNLM with a source context window. Our model is purely lexi- calized and can be integrated into any MT decoder. We also present several varia- tions of the NNJM which provide signif- icant additive improvements. Although the model is quite simple, it yields strong empirical results. On the NIST OpenMT12 Arabic-English condi- tion, the NNJM features produce a gain of +3.0 BLEU on top of a powerful, feature- rich baseline which already includes a target-only NNLM. The NNJM features also produce a gain of +6.3 BLEU on top of a simpler baseline equivalent to Chi- ang’s (2007) original Hiero implementa- tion. Additionally, we describe two novel tech- niques for overcoming the historically high cost of using NNLM-style models in MT decoding. These techniques speed up NNJM computation by a factor of 10,000x, making the model as fast as a standard back-off LM. This work was supported by DARPA/I2O Contract No. HR0011-12-C-0014 under the BOLT program (Approved for Public Release, Distribution Unlimited). The views, opin- ions, and/or findings contained in this article are those of the author and should not be interpreted as representing the of- ficial views or policies, either expressed or implied, of the Defense Advanced Research Projects Agency or the Depart- ment of Defense. 1 Introduction In recent years, neural network models have be- come increasingly popular in NLP. Initially, these models were primarily used to create n-gram neu- ral network language models (NNLMs) for speech recognition and machine translation (Bengio et al., 2003; Schwenk, 2010). They have since been ex- tended to translation modeling, parsing, and many other NLP tasks. In this paper we use a basic neural network ar- chitecture and a lexicalized probability model to create a powerful MT decoding feature. Specifi- cally, we introduce a novel formulation for a neu- ral network joint model (NNJM), which augments an n-gram target language model with an m-word source window. Unlike previous approaches to joint modeling (Le et al., 2012), our feature can be easily integrated into any statistical machine trans- lation (SMT) decoder, which leads to substantially larger improvements than k-best rescoring only. Additionally, we present several variations of this model which provide significant additive BLEU gains. We also present a novel technique for training the neural network to be self-normalized, which avoids the costly step of posteriorizing over the entire vocabulary in decoding. When used in con- junction with a pre-computed hidden layer, these techniques speed up NNJM computation by …

Content truncated to 500 words