Machine translation evaluation made fuzzier: a study on post-editing productivity and evaluation metrics in commercial settings

Machine translation evaluation made fuzzier: A study on post-editing productivity and evaluation metrics in commercial settings Carla Parra Escart´ın carla.parra@hermestrans.com Manuel Arcedillo manuel.arcedillo@hermestrans.com Hermes Traducciones C/ C´olquide 6, portal 2, 3. I 28230 Las Rozas, Madrid, Spain Abstract In this paper, we report on an experiment carried out in the context of a translation company. Ten translators, with diverse degrees of experience in translation and machine translation post- editing (MTPE), were assigned the same task, involving translation from scratch, fuzzy-match post-editing, and MTPE. We evaluate the MT output using traditional evaluation metrics such as BLEU and TER, correlate these measures with productivity values and study whether a fuzzy score stands up against them. Our main goal was to evaluate whether fuzzy scores can be used for evaluating MTPE, thus incorporating its familiarity and TM matching analogies to an MTPE workflow. The results of our experiment seem to support this hypothesis. 1 Introduction In the last years, machine translation post-editing (MTPE) tasks have become ever more com- mon in the translation industry. Translators and translation companies of all sizes use machine translation (MT) to increase their productivity and reduce production costs. However, it is still unclear how to assess MT output in order to verify that those goals have been met and, if appli- cable, determine a fair compensation for the post-editor. Traditional automatic evaluation metrics such as BLEU (Papineni et al., 2001) and TER (Snover et al., 2006) have shortcomings such as unproven correlation with productivity gains (Callison-Burch et al., 2006), technical difficulties for their estimation by general users and lack of intuitiveness. Meanwhile, the translation industry has a well-established way of evaluating text similarity and establishing discount rates: translation memory (TM) fuzzy match scores (FMS). Based on text similarity with segments stored in the TM, each sentence to be translated receives a fuzzy score, ranging from 0% (no similar sentence was found in the TM) to 100% (an exact match was found). In the translation industry, TM matches above 75% are normally assigned a rate discount, while those segments below this threshold are paid the full rate follow- ing the general assumption that they do not yield any productivity increase.1 Since translators and other parties involved in a translation project are familiar with this system and accept it as a valid business model, we designed an experiment to verify whether target-side FMS stand up 1This is an industry general practice. Plitt and Masselot (2010) also report that in a typical localization scenario TM matches below 75% are considered no-matches. However, lack of research and the peculiarities of fuzzy score calculation in each tool may allow for variations of this model. Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 131 against traditional methods of MTPE evaluation. In the experiment, ten professional translators with diverse experience in the industry were assigned the same file to translate, involving translation from scratch, TM match post-editing, and MTPE. We recorded the time spent by …

Content truncated to 500 words