A Reading Comprehension Corpus for Machine Translation Evaluation

A Reading Comprehension Corpus for Machine Translation Evaluation Carolina Scarton and Lucia Specia Department of Computer Science, University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK {c.scarton, l.specia}@sheffield.ac.uk Abstract Effectively assessing Natural Language Processing output tasks is a challenge for research in the area. In the case of Machine Translation (MT), automatic metrics are usually preferred over human evaluation, given time and budget constraints. However, traditional automatic metrics (such as BLEU) are not reliable for absolute quality assessment of documents, often producing similar scores for documents translated by the same MT system. For scenarios where absolute labels are necessary for building models, such as document-level Quality Estimation, these metrics can not be fully trusted. In this paper, we introduce a corpus of reading comprehension tests based on machine translated documents, where we evaluate documents based on answers to questions by fluent speakers of the target language. We describe the process of creating such a resource, the experiment design and agreement between the test takers. Finally, we discuss ways to convert the reading comprehension test into document-level quality scores. Keywords: Machine Translation, Reading Comprehension, Quality Estimation 1. Introduction Evaluating Machine Translation (MT) systems outputs is a challenging task. Whether the evaluation goal is to compare MT systems, to inform end-users or to assist in the trans- lation process (such as in post-editing), appropriate evalu- ation methods and metrics need to be applied in order to provide reliable assessments. Automatic metrics that contrast system outputs against ref- erence translations, such as BLEU (Papineni et al., 2002), are widely explored to compare MT systems and measure the progress of a given MT system over time. Quality Es- timation (QE) is a different evaluation method that pro- vides a quality prediction for new, unseen machine trans- lated texts, without relying on reference translations (Blatz et al., 2004; Specia et al., 2009). These metrics are use- ful to inform end-users and post-editors. Most work in QE focuses on sentence-level and word-level prediction (Callison-Burch et al., 2012; Bojar et al., 2013; Bojar et al., 2014; Bojar et al., 2015). Estimation of quality at sentence and word levels are probably the most useful types of pre- diction for post-editing, since post-editors can benefit from smaller parts of the document that are already acceptable, instead of relying on a single quality score for the entire document. On the other hand, document-level QE can be desirable for applications aimed at other types of end-users (such as gisting ) and where fully automated MT is needed (e.g. because the amount of data is unfeasible for human post-editing). While for sentence and word-level QE several quality la- bels have been proposed so far (e.g. HTER (Snover et al., 2006), likert), there is a lack of studies in quality labels for document-level. Previous work use BLEU-style met- rics as labels (Scarton and Specia, 2014; Soricut and Echi- habi, 2010; Scarton, 2015) . The WMT15 QE shared task also followed this approach, using METEOR (Banerjee and Lavie, 2005) for a paragraph-level …

Content truncated to 500 words