Findings of the WMT 2020 Shared Task on Automatic Post-Editing

Proceedings of the 5th Conference on Machine Translation (WMT), pages 646–659 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 646 Findings of the WMT 2020 Shared Task on Automatic Post-Editing Rajen Chatterjee(1), Markus Freitag(2), Matteo Negri(3), Marco Turchi(3) (1) Apple Inc., Cupertino, CA, USA (2) Google Research, Mountain View, CA, USA (3) Fondazione Bruno Kessler, Trento, Italy Abstract We present the results of the 6th round of the WMT task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a “black-box” machine trans- lation system by learning from existing hu- man corrections of different sentences. This year, the challenge consisted of fixing the er- rors present in English Wikipedia pages trans- lated into German and Chinese by state-of- the-art, not domain-adapted neural MT (NMT) systems unknown to participants. Six teams participated in the English-German task, sub- mitting a total of 11 runs. Two teams par- ticipated in the English-Chinese task submit- ting 2 runs each. Due to i) the different source/domain of data compared to the past (Wikipedia vs Information Technology), ii) the different quality of the initial translations to be corrected and iii) the introduction of a new language pair (English-Chinese), this year’s results are not directly comparable with last year’s round. However, on both language di- rections, participants’ submissions show con- siderable improvements over the baseline re- sults. On English-German, the top-ranked sys- tem improves over the baseline by -11.35 TER and +16.68 BLEU points, while on English- Chinese the improvements are respectively up to -12.13 TER and +14.57 BLEU points. Overall, coherent gains are also highlighted by the outcomes of human evaluation, which con- firms the effectiveness of APE to improve MT quality, especially in the new generic domain selected for this year’s round. 1 Introduction MT Automatic Post-Editing (APE) is the task of automatically correcting errors in a machine- translated text. As pointed out by (Chatterjee et al., 2015), from the application point of view, the task is motivated by its possible uses to: • Improve MT output by exploiting informa- tion unavailable to the decoder, or by per- forming deeper text analysis that is too ex- pensive at the decoding stage; • Cope with systematic errors of an MT system whose decoding process is not accessible; • Provide professional translators with im- proved MT output quality to reduce (human) post-editing effort; • Adapt the output of a general-purpose MT system to the lexicon/style requested in a spe- cific application domain. In its 6th round, the APE shared task organized within the WMT Conference on Machine Trans- lation kept the same overall evaluation setting of the previous five rounds. Specifically, the partic- ipating systems had to automatically correct the output of an unknown “black box” (neural) MT system by learning from training data containing human revisions of translations produced by the same system. This year, the task focused on two language pairs: English-German and English-Chinese. The former has been part of the APE evaluation cam- paigns since 2016 (Bojar et al., …

Content truncated to 500 words