Data enhancement and selection strategies for the word-level Quality Estimation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 330–335, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Data Enhancement and Selection Strategies for the Word-level Quality Estimation Varvara Logacheva§, Chris Hokamp†, Lucia Specia§ §Department of Computer Science, University of Sheffield, UK {v.logacheva,l.specia}@sheffield.ac.uk †CNGL Centre for Global Intelligent Content Dublin City University, Ireland chokamp@computing.dcu.ie Abstract This paper describes the DCU-SHEFF word-level Quality Estimation (QE) sys- tem submitted to the QE shared task at WMT15. Starting from a baseline set of features and a CRF algorithm to learn a sequence tagging model, we propose im- provements in two ways: (i) by filtering out the training sentences containing too few errors, and (ii) by adding incomplete sequences to the training data to enrich the model with new information. We also ex- periment with considering the task as a classification problem, and report results using a subset of the features with Random Forest classifiers. 1 Introduction The WMT shared task on Quality estimation (QE) for Machine Translation (MT) has included the sub-task on the QE at the word level since the year 2013. The goal of this task is to assign a quality label to each word of an automatically translated sentence without using its reference translations. The set of possible output labels can vary. Labels can specify the edit action which should be per- formed on the word in order to improve the sen- tence (substitution, deletion, insertion) — these la- bels were used in the WMT13 QE task (Bojar et al., 2013). Labels can be further refined to spec- ify the type of error: grammar error, wrong termi- nology, untranslated word, etc., motivated by the MQM error typology 1 — this tagging was used in last year’s task (Bojar et al., 2014). In both cases, tags can be generalised to a binary label, “GOOD” or “BAD”, indicating whether or not the word is correct. 1http://www.qt21.eu/launchpad/content/ multidimensional-quality-metrics This year, the word-level QE task (Task 2 in WMT15 QE shared task2) consists in assigning only a binary label (“GOOD” or “BAD”) to every word in automatically translated sentences — that is, to identify if a word is suitable for this sentence or should be modified. The possible errors are sub- stitution (word replacement) or insertion. This for- mulation of the task cannot detect deletions in the MT hypothesis, because there is a one-to-one cor- respondence between tokens in the hypothesis and output tags. The data for the word-level QE task was pro- duced for one translation direction, namely from English into Spanish. The training, development and test datasets have been translated automati- cally with an online statistical MT system, and then post-edited by human translators. Besides the datasets themselves, baseline feature sets were provided. The suggested baseline training model is conditional random fields (CRF) (Lafferty et al., 2001), which is one of the most widely used techniques for sequence labelling. The baseline tagging for this task was done with CRF model trained using CRF++ tool3. Our system uses the baseline …

Content truncated to 500 words