Machine Translation Evaluation using Recurrent Neural Networks

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 380–384, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Machine Translation Evaluation using Recurrent Neural Networks Rohit Gupta1, Constantin Or˘asan1, Josef van Genabith2 1Research Group in Computational Linguistics, University of Wolverhampton, UK 2Saarland University and German Research Center for Artificial Intelligence (DFKI), Germany {r.gupta, c.orasan}@wlv.ac.uk josef.van genabith@dfki.de Abstract This paper presents our metric (UoW- LSTM) submitted in the WMT-15 met- rics task. Many state-of-the-art Machine Translation (MT) evaluation metrics are complex, involve extensive external re- sources (e.g. for paraphrasing) and require tuning to achieve the best results. We use a metric based on dense vector spaces and Long Short Term Memory (LSTM) networks, which are types of Recurrent Neural Networks (RNNs). For WMT- 15 our new metric is the best performing metric overall according to Spearman and Pearson (Pre-TrueSkill) and second best according to Pearson (TrueSkill) system level correlation. 1 Introduction Deep learning approaches have turned out to be successful in many NLP applications such as para- phrasing (Mikolov et al., 2013b; Socher et al., 2011), sentiment analysis (Socher et al., 2013b), parsing (Socher et al., 2013a) and machine trans- lation (Mikolov et al., 2013a). While dense vec- tor space representations such as those obtained through Deep Neural Networks (DNNs) or Re- current Neural Networks (RNNs) are able to cap- ture semantic similarity for words (Mikolov et al., 2013b), segments (Socher et al., 2011) and documents (Le and Mikolov, 2014) naturally, tra- ditional measures can only achieve this using re- sources like WordNet and paraphrase databases. This paper presents a novel, efficient and com- pact MT evaluation measure based on RNNs. Our metric (Gupta et al., 2015) is simple in the sense that it does not require much machinery and re- sources apart from the dense word vectors. This cannot be said of most of the state-of-the-art MT evaluation metrics, which tend to be complex and require extensive feature engineering. Our met- ric is based on RNNs and particularly on Tree Long Short Term Memory (Tree-LSTM) networks (Tai et al., 2015). LSTM is a sequence learning technique which uses a memory cell to preserve a state over a long period of time. This enables distributed representations of sentences using dis- tributed representations of words. Tree-LSTM (Tai et al., 2015) is a recent approach, which is an extension of the simple LSTM framework (Hochreiter and Schmidhuber, 1997; Zaremba and Sutskever, 2014). 2 Related Work Many metrics have been proposed for MT eval- uation. Earlier popular metrics are based on n- gram counts (e.g. BLEU (Papineni et al., 2002) and NIST (Doddington, 2002)) or word error rate. Other popular metrics like METEOR (Denkowski and Lavie, 2014) and TERp (Snover et al., 2008) also use external resources like WordNet and para- phrase databases. However, system-level cor- relation with human judgements for these met- rics remains below 0.90 Pearson correlation co- efficient (as per WMT-14 results, BLEU-0.888, NIST-0.867, METEOR-0.829, TER-0.826, WER- 0.821). Recent best performing metrics in the WMT- 14 metric shared …

Content truncated to 500 words