Parmesan: Meteor without Paraphrases with Paraphrased References

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 355–361, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Parmesan: Meteor without Paraphrases with Paraphrased References Petra Baranˇc´ıkov´a Institute of Formal and Applied Linguistics Charles University in Prague, Faculty of Mathematics and Physics Malostransk´e n´amˇest´ı 25, Prague, Czech Republic barancikova@ufal.mff.cuni.cz Abstract This paper describes Parmesan, our sub- mission to the 2014 Workshop on Sta- tistical Machine Translation (WMT) met- rics task for evaluation English-to-Czech translation. We show that the Czech Me- teor Paraphrase tables are so noisy that they actually can harm the performance of the metric. However, they can be very useful after extensive filtering in targeted paraphrasing of Czech reference sentences prior to the evaluation. Parmesan first per- forms targeted paraphrasing of reference sentences, then it computes the Meteor score using only the exact match on these new reference sentences. It shows sig- nificantly higher correlation with human judgment than Meteor on the WMT12 and WMT13 data. 1 Introduction The metric for automatic evaluation of machine translation (MT) Meteor1 (Denkowski and Lavie, 2011) has shown high correlation with human judgment since its appearance. It outperforms tra- ditional metrics like BLEU (Papineni et al., 2002) or NIST (Doddington, 2002) as it explicitly ad- dresses their weaknesses – it takes into account re- call, distinguishes between functional and content words, allows language-specific tuning of param- eters and many others. Another important advantage of Meteor is that it supports not only exact word matches between a hypothesis and its corresponding reference sen- tence, but also matches on the level of stems, syn- onyms and paraphrases. The Meteor Paraphrase tables (Denkowski and Lavie, 2010) were created automatically using the pivot method (Bannard and Callison-Burch, 2005) for six languages. 1We use the the version 1.4., which was recently outdated as the new version 1.5. was released for WMT14 The basic setting of Meteor for evaluation of Czech sentences offers two levels of matches - exact and paraphrase. In this paper, we show the impact of the quality of paraphrases on the perfor- mance of Meteor. We demonstrate that the Czech Meteor Paraphrase tables are full of noise and their addition to the metric worsens its correlation with human judgment. However, they can be very use- ful (after extensive filtering) in creating new refer- ence sentences by targeted paraphrasing. Parmesan2 starts with a simple greedy algo- rithm for substitution of synonymous words from a hypothesis in its corresponding reference sen- tence. Further, we apply Depfix (Rosa et al., 2012) to fix grammar errors that might arise by the sub- stitutions. Our method is independent of the evaluation metric used. In this paper, we use Meteor for its consistently high correlation with human judg- ment and we attempt to tune it further by mod- ifying its paraphrase tables. We show that re- ducing the size of the Meteor Paraphrase tables is very beneficial. On the WMT12 and WMT13 data, the Meteor scores computed using only the exact match on our new references …

Content truncated to 500 words