USHEF and USAAR-USHEF participation in the WMT15 QE shared task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 336–341, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. USHEF and USAAR-USHEF Participation in the WMT15 Quality Estimation Shared Task Carolina Scarton1, Liling Tan2 and Lucia Specia1 1University of Sheffield, 211 Portobello Street, Sheffield, UK 2Saarland University, Campus A2.2, Saarbr¨ucken, Germany {c.scarton,l.specia}@sheffield.ac.uk alvations@gmail.com Abstract We present the results of the USHEF and USAAR-USHEF submissions for the WMT15 shared task on document-level quality estimation. The USHEF sub- missions explored several document and discourse-aware features. The USAAR- USHEF submissions used an exhaustive search approach to select the best features from the official baseline. Results show slight improvements over the baseline with the use of discourse features. More inter- estingly, we found that a model of compa- rable performance can be built with only three features selected by the exhaustive search procedure. 1 Introduction Evaluating the quality of Machine Translation (MT) systems outputs is a challenging topic. Sev- eral metrics have been proposed so far comparing the MT outputs to human translations (references) in terms of ngrams matches (such as BLEU (Pa- pineni et al., 2002)) or error rates (such as TER (Snover et al., 2006)). However, in some scenar- ios, human references are not available. For ex- ample, the use of machine translation in a work- flow where good enough translations are given to humans for post-editing. Another example is ma- chine translation for gisting by users of online sys- tems. Quality Estimation (QE) approaches aim to pre- dict the quality of MT outputs without relying on human references (Blatz et al., 2004; Specia et al., 2009). Features from source (original document) and target (MT outputs) and, when available, from the MT system are used to train supervised ma- chine learning models (classifiers or regressors). A number of data points need to be annotated for quality (by humans or automatically) for training, using a given quality metric. Most QE research is done at sentence level. This task has been a track at WMT shared task for the last four years (Callison-Burch et al., 2012; Bojar et al., 2013; Bojar et al., 2014). In addi- tion to sentence level, the current edition offers for the first time a track on paragraph-level QE. Ex- ploring quality beyond sentence level is interesting for completely automatic translation applications, i.e. without human review. For instance, consider a user looking for information on a product that has several reviews automatically translated into his/her language. This user have no knowledge about the source language. To ensure that the main message of the review is preserved, for this user the quality of each word or sentence individually is not as important as the quality of the review as a whole. Therefore, predicting the quality of the whole document (or paragraph, considering para- graph as short documents) becomes necessary. This paper presents the University of Sheffield (USHEF) and University of Saarland (USAAR) submissions to the Task 3 of the WMT15 QE shared task: paragraph-level scoring and ranking. We submitted …

Content truncated to 500 words