Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and its relation with Direct Assessment

Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and its relation with Direct Assessment Luisa Bentivogli(1), Mauro Cettolo(1), Marcello Federico(2), Christian Federmann(3) (1) FBK - Trento, Italy (2) Amazon AI - East Palo Alto, CA, USA (3) Microsoft Cloud+AI - Redmond, WA, USA [bentivo|cettolo]@fbk.eu marcfede@amazon.com chrife@microsoft.com Abstract In this paper we present an analysis of the two most promi- nent methodologies used for the human evaluation of MT quality, namely evaluation based on Post-Editing (PE) and evaluation based on Direct Assessment (DA). To this pur- pose, we exploit a publicly available large dataset containing both types of evaluations. We first focus on PE and investi- gate how sensitive TER-based evaluation is to the type and number of references used. Then, we carry out a compara- tive analysis of PE and DA to investigate the extent to which the evaluation results obtained by methodologies addressing different human perspectives are similar. This comparison sheds light not only on PE but also on the so-called reference bias related to monolingual DA. Also, we analyze if and how the two methodologies can complement each other’s weak- nesses. 1. Introduction The evaluation of machine translation (MT) is of crucial im- portance and has a long research history. Both human and automatic evaluation have been explored extensively within the MT community, in the effort to find more and more suit- able, efficient and reliable methods and metrics. Automatic metrics play a central role in the progress of the field and the improvement of MT quality over time. However, they rep- resent a proxy for human evaluation which – despite being costly and time-consuming – is to be considered primary. Among the various human evaluation methods that have been devised and tested along the years, currently two ap- proaches have become well-established standards in the field, namely evaluation based on Post-Editing (PE) and evaluation based on Direct Assessment (DA). In the PE-based evaluation, the MT outputs are post- edited, i.e. manually corrected, according to the source sen- tence (bilingual PE) or to an existing reference translation (2) Work conducted while this author was at FBK. (monolingual PE). The original MT outputs are then eval- uated against their post-edited versions through TER-based automatic metrics [1]. Relying on the post-edit instead of an independently created reference translation ensures that only true errors in the MT output are counted, and not those differ- ences due to linguistic variation, which are accounted for by post-editors. PE has become the standard evaluation metric for the yearly evaluation campaign of the International Work- shop of Spoken Language Translation since 2013 (IWSLT- 2013) and is described in detail in [2]. The DA-based evaluation [3] consists of collecting hu- man assessments of translation quality for single MT sys- tems. Assessors see a candidate translation and a correspond- ing translation hint (e.g. the source text, a reference transla- tion, or multimodal content) and are asked to assign a quality score from 0 to 100. DA has become the standard evalua- tion metric for …

Content truncated to 500 words