A Post-Editing Dataset in the Legal Domain: Do we Underestimate Neural Machine Translation Quality?

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3692–3697 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3692 A Post-Editing Dataset in the Legal Domain: Do we Underestimate Neural Machine Translation Quality? Julia Ive1, Lucia Specia1,2, Sara Szoc3, Tom Vanallemeersch3, Joachim Van den Bogaert3, Eduardo Farah4, Christine Maroti4, Artur Ventura4, Maxim Khalilov4 1 Department of Computer Science, University of Sheffield, UK, 2 Department of Computing, Imperial College London, UK, 3 CrossLang, Belgium 4 Unbabel, Portugal j.ive@sheffield.ac.uk, l.specia@imperial.ac.uk first.lastname@crosslang.com eduardo.farah@unbabel.com, christine@unbabel.com, artur@unbabel.com, maxim@unbabel.com Abstract We introduce a machine translation dataset for three pairs of languages in the legal domain with post-edited high-quality neural machine translation and independent human references. The data was collected as part of the EU APE-QUEST project and comprises crawled content from EU websites with translation from English into three European languages: Dutch, French and Portuguese. Altogether, the data consists of around 31K tuples including a source sentence, the respective machine translation by a neural machine translation system, a post-edited version of such translation by a professional translator, and – where available – the original reference translation crawled from parallel language websites. We describe the data collection process, provide an analysis of the resulting post-edits and benchmark the data using state-of-the-art quality estimation and automatic post-editing models. One interesting by-product of our post-editing analysis suggests that neural systems built with publicly available general domain data can provide high-quality translations, even though comparison to human references suggests that this quality is quite low. This makes our dataset a suitable candidate to test evaluation metrics. The data is freely available as an ELRC-SHARE resource. Keywords: Machine Translation, Post-editing Dataset, Legal Domain, Automatic Post-Editing, Quality Estimation 1. Introduction Current state-of-the-art (SOTA) in Neural Machine Trans- lation (NMT) has reached remarkable progress. This has also pushed the boundaries of manual and automatic evalu- ation procedures relying on human reference translations. For example, recent studies have shown that reference translations are often judged by humans as having lower translation quality than top NMT systems (Hassan et al., 2018), with follow-up studies showing that this is partly due to the limited quality of the human translations (Toral et al., 2018). Even though problems with independently collected (single) human reference translations for evaluation have been highlighted in the past (Fomicheva and Specia, 2016), this practice is more questionable with high quality NMT systems. In these conditions, metrics based on human post- editing of the machine translations become particularly im- portant. This feedback can be used to assess MT quality directly, as well as to build and benchmark metrics for the automatic evaluation of Machine Translation (MT) output, and to build Quality Estimation (QE) and Automatic PE (APE) models. We describe a machine translation dataset in the legal do- main resulting from activities performed in the framework of the APE-QUEST project (http://ape-quest.eu). This project aims to integrate MT, QE and APE. The dataset focuses on the areas of online dispute resolution (ODR), procurement and justice. It …

Content truncated to 500 words