ReVal: A Simple and Effective Machine Translation Evaluation Metric Based on Recurrent Neural Networks

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1066–1072, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. ReVal: A Simple and Effective Machine Translation Evaluation Metric Based on Recurrent Neural Networks Rohit Gupta1, Constantin Or˘asan1, Josef van Genabith2 1Research Group in Computational Linguistics, University of Wolverhampton, UK 2Saarland University and German Research Center for Artificial Intelligence (DFKI), Germany {r.gupta, c.orasan}@wlv.ac.uk josef.van genabith@dfki.de Abstract Many state-of-the-art Machine Translation (MT) evaluation metrics are complex, in- volve extensive external resources (e.g. for paraphrasing) and require tuning to achieve best results. We present a simple alternative approach based on dense vec- tor spaces and recurrent neural networks (RNNs), in particular Long Short Term Memory (LSTM) networks. For WMT-14, our new metric scores best for two out of five language pairs, and overall best and second best on all language pairs, using Spearman and Pearson correlation, respec- tively. We also show how training data is computed automatically from WMT ranks data. 1 Introduction Deep learning approaches have turned out to be successful in many NLP applications such as para- phrasing (Mikolov et al., 2013b; Socher et al., 2011), sentiment analysis (Socher et al., 2013b), parsing (Socher et al., 2013a) and machine trans- lation (Mikolov et al., 2013a). While dense vec- tor space representations such as those obtained through Deep Neural Networks (DNNs) or RNNs are able to capture semantic similarity for words (Mikolov et al., 2013b), segments (Socher et al., 2011) and documents (Le and Mikolov, 2014) naturally, traditional MT evaluation metrics can only achieve this using resources like WordNet and paraphrase databases. This paper presents a novel, efficient and compact MT evaluation mea- sure based on RNNs. Our metric is simple in the sense that it does not require much machinery and resources apart from the dense word vectors. This cannot be said of most of the state-of-the-art MT evaluation metrics, which tend to be complex and require extensive feature engineering. Our metric is based on RNNs and particularly on Tree Long Short Term Memory (Tree-LSTM) networks (Tai et al., 2015). LSTM (Hochreiter and Schmidhu- ber, 1997) is a sequence learning technique which uses a memory cell to preserve a state over a long period of time. This enables distributed represen- tations of sentences using distributed representa- tions of words. Tree-LSTM is a recent approach, which is an extension of the simple LSTM frame- work (Zaremba and Sutskever, 2014). To provide the required training data, we also show how to automatically convert the WMT-13 (Bojar et al., 2013) human evaluation rankings into similarity scores between the reference and the translation. Our metric including training data is available at https://github.com/rohitguptacs/ReVal. 2 Related Work Many metrics have been proposed for MT eval- uation. Earlier popular metrics are based on n- gram counts (e.g. BLEU (Papineni et al., 2002) and NIST (Doddington, 2002)) or word error rate. Other popular metrics like METEOR (Denkowski and Lavie, 2014) and TERp (Snover et al., 2008) also use external resources like WordNet and para- …

Content truncated to 500 words