Manual and Automatic Paraphrases for MT Evaluation

Manual and Automatic Paraphrases for MT Evaluation Aleˇs Tamchyna, Petra Baranˇc´ıkov´a Charles University in Prague, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics {tamchyna,barancikova}@ufal.mff.cuni.cz Abstract Paraphrasing of reference translations has been shown to improve the correlation with human judgements in automatic evaluation of machine translation (MT) outputs. In this work, we present a new dataset for evaluating English-Czech translation based on automatic paraphrases. We compare this dataset with an existing set of manually created paraphrases and find that even automatic paraphrases can improve MT evaluation. We have also propose and evaluate several criteria for selecting suitable reference translations from a larger set. Keywords: machine translation, automatic evaluation, paraphrasing 1. Introduction In a language, an idea might be expressed in an immense number of ways. This poses a serious issue for many lin- guistic tasks and particularly for machine translation (MT) evaluation. In automatic machine translation evaluation, outputs of an MT system are compared to a reference translation, i.e. translation provided by a human translator. Due to the ex- cessive amount of possible translations, metrics for this au- tomatic comparison tend not to reflect human judgement very well. These metrics incline to perform better if there are more reference sentences available. The pioneer metric BLEU (Papineni et al., 2002) was originally designed to work with four reference sentences. However, obtaining reference sentences is labour inten- sive1 and expensive. MT competitions, such as the Work- shop on Statistical Machine Translation (WMT), use only one reference sentence for the automatic evaluation. This is still a considerable financial amount2 given the number of languages and sentences. Our motivation in this paper is to build on the work of hu- man translators and to generate more reference translations automatically by paraphrasing the original manual transla- tions. Our overall goal is to provide more robust evaluation of MT systems in this way. There is evidence that paraphrasing of reference transla- tions improves MT evaluation: in (Bojar et al., 2013b), human annotators were asked to create “all possible” paraphrases of 50 sentences from the WMT11 evaluation dataset (Callison-Burch et al., 2011). The authors show that correlation of MT evaluation metrics with human judgement increases with adding reference paraphrases. In this work, we propose a simple approach to creating 1E.g. at the Linguistic Data Consortium, reference translation production is a complicated process which involves professional translation agencies, elaborate guidelines and thorough quality control (Strassel et al., 2006). 2For example, in the year 2009, the total cost of creating the test set was approximately $39,800 USD. It consists of 3027 sentences in seven European languages, which corresponds to slightly more than $0.08 USD/word (Bloodgood and Callison- Burch, 2010). such a dataset automatically and compare the usefulness of automatic and manual paraphrases for MT evaluation. Once such a large set of reference translations is created, it can simply be utilized within a standard metric such as BLEU. However, the number of possible paraphrased translations for a sentence can often be very high, which may result in unnecessarily …

Content truncated to 500 words