Investigating Continuous Space Language Models for Machine Translation Quality Estimation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1073–1078, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Investigating Continuous Space Language Models for Machine Translation Quality Estimation Kashif Shah§, Raymond W. M. Ng§, Fethi Bougares†, Lucia Specia§ §Department of Computer Science, University of Sheffield, UK {kashif.shah, wm.ng, l.specia}@sheffield.ac.uk †LIUM, University of Le Mans, France fethi.bougares@lium.univ-lemans.fr Abstract We present novel features designed with a deep neural network for Machine Trans- lation (MT) Quality Estimation (QE). The features are learned with a Continuous Space Language Model to estimate the probabilities of the source and target seg- ments. These new features, along with standard MT system-independent features, are benchmarked on a series of datasets with various quality labels, including post- editing effort, human translation edit rate, post-editing time and METEOR. Results show significant improvements in predic- tion over the baseline, as well as over sys- tems trained on state of the art feature sets for all datasets. More notably, the addition of the newly proposed features improves over the best QE systems in WMT12 and WMT14 by a significant margin. 1 Introduction Quality Estimation (QE) is concerned with pre- dicting the quality of Machine Translation (MT) output without reference translations. QE is ad- dressed with various features indicating fluency, adequacy and complexity of the translation pair. These features are used by a machine learning al- gorithm along with quality labels given by humans to learn models to predict the quality of unseen translations. A variety of features play a key role in QE. A wide range of features from source segments and their translated segments, extracted with the help of external resources and tools, have been proposed. These go from simple, language- independent features, to advanced, linguistically motivated features. They include features that summarise how the MT systems generate transla- tions, as well as features that are oblivious to the systems. The majority of the features in the lit- erature are extracted from each sentence pair in isolation, ignoring the context of the text. QE performance usually differs depending on the lan- guage pair, the specific quality score being opti- mised (e.g., post-editing time vs translation ad- equacy) and the feature set. Features based on n-gram language models, despite their simplicity, are among those with the best performance in most QE tasks (Shah et al., 2013b). However, they may not generalise well due to the underlying discrete nature of words in n-gram modelling. Continuous Space Language Models (CSLM), on the other hand, have shown their potential to capture long distance dependencies among words (Schwenk, 2012; Mikolov et al., 2013). The assumption of these models is that semantically or grammatically related words are mapped to simi- lar geometric locations in a high-dimensional con- tinuous space. The probability distribution is thus much smoother and therefore the model has a bet- ter generalisation power on unseen events. The representations are learned in a continuous space to estimate the probabilities using neural networks with single (called shallow networks) or multi- …

Content truncated to 500 words