Why Predicting Post-Edition is so Hard? Failure Analysis of LIMSI Submission to the APE Shared Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 222–227, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Why Predicting Post-Edition is so Hard? Failure Analysis of LIMSI Submission to the APE Shared Task Guillaume Wisniewski and Nicolas P´echeux and Franc¸ois Yvon Universit´e Paris Sud and LIMSI-CNRS 91 403 ORSAY CEDEX, France {wisniews,pecheux,yvon}@limsi.fr Abstract This paper describes the two systems sub- mitted by LIMSI to the WMT’15 Shared Task on Automatic Post-Editing. The first one relies on a reformulation of the APE task as a Machine Translation task; the second implements a simple rule-based approach. Neither of these two systems manage to improve the automatic transla- tion. We show, by carefully analyzing the failure of our systems that this counter- performance mainly results from the in- consistency in the annotations. 1 Introduction This paper describes LIMSI submission to the WMT’15 Shared Task on Automatic Post-Editing (APE). This task aims at automatically correcting errors produced by an unknown Machine Transla- tion (MT) system by learning from human post- editions. For the first edition of this Shared Task we have submitted two APE systems. The first one, de- scribed in Section 3, is based on the approach of Simard et al. (2007) and considers the APE task as the automatic translation between a transla- tion hypothesis and its post-edition. This straight- forward approach does not succeed in improving translation quality. To understand the reasons of this failure, we present, in Section 4 a detailed analysis of the training data that highlights some of the difficulties of training an APE system. The second submitted system implements a se- ries of sieves, applying, each, a simple post- editing rule. The definition of these rules is based on our analysis of the most frequent error correc- tions. Experiments with this approach (Section 5) show that this system also hurts translation quality. However, analyzing its failures allows us to show that the main difficulties in correcting MT errors result from the inconsistency between the differ- ent post-editions. 2 Data Preprocessing The Shared Task organizers provide training and development data that consist of respectively 11,272 and 1,000 examples. Each example is made of an English source sentence, its automatic translation in Spanish by an unknown MT sys- tem and a human revision of this translation. All sentences are tokenized. There are, on average, 22.88 words in each post-edition, the longest post- edition having 199 words and the shortest 3. In a first pre-processing step we have removed all examples for which the ratio between the length of the automatic translation and the length of the corresponding post-edition was higher than 1.2 or lower than 0.8. As shown in Table 1, these ex- amples correspond mainly to errors in sentence boundaries or to ‘over-translation’ (e.g. when the post-editor added the translated title in the third example of Table 1), that could have a negative impact on the training of an APE system. At the end, the training set we used in all our experiments is …

Content truncated to 500 words