SHEF-NN: Translation Quality Estimation with Neural Networks

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 342–347, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. SHEF-NN: Translation Quality Estimation with Neural Networks Kashif Shah§, Varvara Logacheva§, Gustavo Henrique Paetzold§, Frederic Blain§ Daniel Beck§, Fethi Bougares†, Lucia Specia§ §Department of Computer Science, University of Sheffield, UK {kashif.shah,v.logacheva,ghpaetzold1,f.blain,debeck1,l.specia} @sheffield.ac.uk †LIUM, University of Le Mans, France fethi.bougares@lium.univ-lemans.fr Abstract We describe our systems for Tasks 1 and 2 of the WMT15 Shared Task on Qual- ity Estimation. Our submissions use (i) a continuous space language model to ex- tract additional features for Task 1 (SHEF- GP, SHEF-SVM), (ii) a continuous bag- of-words model to produce word embed- dings as features for Task 2 (SHEF-W2V) and (iii) a combination of features pro- duced by QuEst++ and a feature produced with word embedding models (SHEF- QuEst++). Our systems outperform the baseline as well as many other submis- sions. The results are especially encour- aging for Task 2, where our best perform- ing system (SHEF-W2V) only uses fea- tures learned in an unsupervised fashion. 1 Introduction Quality Estimation (QE) aims at measuring the quality of the Machine Translation (MT) output without reference translations. Generally, QE is addressed with various features indicating fluency, adequacy and complexity of the source-translation text pair. Such features are then used along with Machine Learning methods in order for models to be learned. Features play a key role in QE. A wide range of features from the source segments and their translations, often processed using external re- sources and tools, have been proposed. These go from simple, language-independent features, to advanced, linguistically motivated features. They include features that rely on information from the MT system that generated the translations, and features that are oblivious to the way translations were produced. This leads to a potential bottle- neck: feature engineering can be time consuming, particularly because the impact of features vary across datasets and language pairs. Also, most features in the literature are extracted from seg- ment pairs in isolation, ignoring contextual clues from other segments in the text. The focus of our contributions this year is to introduce a new set of features which are language-independent, require minimal resources, and can be extracted in unsu- pervised ways with the use of neural networks. Word embeddings have shown their poten- tial in modelling long distance dependencies in data, including syntactic and semantic informa- tion. For instance, neural network language mod- els (Bengio et al., 2003) have been success- fully explored in many problems including Au- tomatic Speech Recognition (Schwenk and Gau- vain, 2005; Schwenk, 2007) and Machine Transla- tion (Schwenk, 2012). While neural network lan- guage models predict the next word given a pre- ceding context, (Mikolov et al., 2013b) proposed a neural network framework to predict the word given the left and right contexts, or to predict the word’s left and right contexts in a given sentence. Recently, it has been shown that these distributed vector representations (or word embeddings) can be exploited across languages to predict …

Content truncated to 500 words