Bilexical Embeddings for Quality Estimation

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 545–550 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Bilexical Embeddings for Quality Estimation Fr´ed´eric Blain, Carolina Scarton and Lucia Specia Department of Computer Science University of Sheffield, UK {f.blain, c.scarton, l.specia}@sheffield.ac.uk Abstract This paper describes the SHEF submis- sions for the three sub-tasks of the Qual- ity Estimation shared task of WMT17, namely: (i) a word-level prediction sys- tem using bilexical embeddings, (ii) a phrase-level labelling approach based on the word-level predictions, (iii) a sentence- level prediction system using word em- beddings and handcrafted baseline fea- tures. Results are promising for the sentence-level approach, but still very pre- liminary for the other two levels. 1 Introduction Quality Estimation (QE) allows the evaluation of Machine Translation (MT) when reference trans- lations are not available. It can be used in vari- ous ways such as in post-editing (PE) to predict whether or not an automatically generated sen- tence is worth publishing, editing or it should be retranslated manually. Word-level predictions can be helpful by highlighting words that cannot be relied upon or should be fixed by post-editors. More recently, QE at phrase-level has emerged as a way of using quality predictions at decoding time in phrase-based Statistical MT (SMT) sys- tems to guide the decoder such as to keep phrases which are predicted as good, and conversely to dis- card those which are predicted as bad (Logacheva, 2017). QE models are built based on a list of features along with a Machine Learning algorithm for ei- ther regression or classification. These features are usually extracted from the source and target texts or from the MT system that generated the transla- tions. Shah et al. (2015) introduced a new set of features extracted using an unsupervised approach with the use of neural network: continuous-space language model features and word embeddings features. In our contribution this year we investigate whether we can go beyond engineered features by learning bilexical operators over distributional representations of words in source-target text pairs. Considering the MT pipeline as a noisy black-box, our motivation is to be able to build QE models to predict if information encoded in the source sentence is preserved in the target sentence after translation. 2 Bilinear Model Madhyastha et al. (2014) propose to use word- level embeddings to predict the strength of differ- ent types of lexical relationships between a pair of words, such as head-modifier relations between noun-adjective pairs. They designed a supervised framework for learning bilexical operators over distributional representations, based on learning bilinear forms W. We adapted their method to predict the strength of relationship between source and target words. This problem is formulated as a log-bilinear model, parametrized with W as fol- lows: Pr(t|s; W) = exp � φ(t)⊤Wφ(s) � � t′∈T exp � φ(t′)⊤Wφ(s) � (1) where φ denotes the word embeddings of any given word in a vocabulary V. The source words s and target words t are respectively taken from …

Content truncated to 500 words