LIG System for Word Level QE task at WMT14

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 335–341, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics LIG System for Word Level QE task at WMT14 Ngoc-Quang Luong Laurent Besacier LIG, Campus de Grenoble 41, Rue des Math´ematiques, UJF - BP53, F-38041 Grenoble Cedex 9, France {ngoc-quang.luong,laurent.besacier,benjamin.lecouteux}@imag.fr Benjamin Lecouteux Abstract This paper describes our Word-level QE system for WMT 2014 shared task on Spanish - English pair. Compared to WMT 2013, this year’s task is different due to the lack of SMT setting information and additional resources. We report how we overcome this challenge to retain most of the important features which performed well last year in our system. Novel features related to the availability of multiple systems output (new point of this year) are also proposed and experimented along with baseline set. The system is optimized by several ways: tuning the classification threshold, combining with WMT 2013 data, and refining using Feature Selection strategy on our development set, before dealing with the test set for submission. 1 Introduction 1.1 Overview of task 2 in WMT14 This year WMT calls for methods which predict the MT output quality at run-time, on both levels: sentence (Task 1) and word (Task 2). Towards a SMT system-independent and widely-applied estimation, MT outputs are collected from multiple translation means (machine and human), therefore all SMT specific settings (and the associated features that could have been extracted from it) become unavailable. This initiative puts more challenges on participants, yet motivates number of SMT-unconventional approaches and inspires the endeavors aiming at an “Evaluation For All”. We focus our effort on Task 2 (Word-level QE), where, unlike in WMT2013, participants are requested to generate prediction labels for words in three variants: • Binary: words are judged as Good (no translation error), or Bad (need for editing). • Level 1: the Good class is kept intact, whereas Bad one is further divided into subcategories: Accuracy issue (the word does not accurately reflect the source text) and Fluency issue (the word does not relate to the form or content of the target text). • Multi-class: more detailed judgement, where the translation errors are further decomposed into 16 labels based on MQM1 metric. 1.2 Related work WMT 2013 witnessed several attempts dealing with this evaluation type in its first launch. Han et al. (2013); Luong et al. (2013) employed the Conditional Random Fields (CRF) (Lafferty et al., 2001) model as their Machine Learning method to address the problem as a sequence labeling task. Meanwhile, Bicici (2013) extended the global learning model by dynamic training with adaptive weight updates in the perceptron training algorithm. As far as prediction indicators are concerned, Bicici (2013) proposed seven word feature types and found among them the “common cover links” (the links that point from the leaf node containing this word to other leaf nodes in the same subtree of the syntactic tree) the most outstanding. Han et al. (2013) focused only on various n-gram combinations of target words. Inheriting …

Content truncated to 500 words