Translation Quality and Productivity: A Study on Rich Morphology Languages

Translation Quality and Productivity: A Study on Rich Morphology Languages Lucia Specia l.specia@sheffield.ac.uk Department of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK Kim Harris kim harris@textform.com text & form GmbH, 10179 Berlin, Germany Fr´ed´eric Blain f.blain@sheffield.ac.uk Department of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK Aljoscha Burchardt aljoscha.burchardt@dfki.de Viviven Macketanz vivien.macketanz@dfki.de Language Technology Lab, DFKI, Alt-Moabit 91c, 10559 Berlin, Germany Inguna Skadin¸a inguna.skadina@tilde.lv Tilde, Vienibas gatve 75a, Riga, LV 1004, Latvia Matteo Negri negri@fbk.eu Marco Turchi turchi@fbk.eu Fondazione Bruno Kessler, Trento, 38100, Italy Abstract This paper introduces a unique large-scale machine translation dataset with various levels of human annotation combined with automatically recorded productivity features such as time and keystroke logging and manual scoring during the annotation process. The data was collected as part of the EU-funded QT21 project and comprises 20,000–45,000 sentences of industry- generated content with translation into English and three morphologically rich languages: English–German/Latvian/Czech and German–English, in either the information technology or life sciences domain. Altogether, the data consists of 176,476 tuples including a source sentence, the respective machine translation by a statistical system (additionally, by a neural system for two language pairs), a post-edited version of such translation by a native-speaking professional translator, an independently created reference translation, and information on post- editing: time, keystrokes, Likert scores, and annotator identifier. A subset of 2,000 sentences from this data per language pair and system type was also manually annotated with translation errors for deeper linguistic analysis. We describe the data collection process, provide a brief analysis of the resulting annotations and discuss the use of the data in quality estimation and automatic post-editing tasks. 1 Introduction Data-driven approaches to machine translation (MT) rely largely on datasets of source sentences and their corresponding translations previously created by humans, so-called parallel corpora. MT systems, be they statistical or neural, are built in static fashion and (if at all) updated from time to time as more translations become available. With the popularisation of post-editing Proceedings of MT Summit XVI, vol.1: Research Track Nagoya, Sep. 18-22, 2017 | p. 55 (PE), a natural question is whether the corrected version of the MT output could be used in feedback loops to improve the current system via model retraining, model tuning or the addition of explicit model components. Additionally, by studying PE data, one can get insights on the errors made by the MT system to try and remedy them in different ways. PE data can also be used to build and benchmark metrics for the automatic evaluation of MT output, as well as quality estimation metrics and automatic PE systems. To facilitate research in these and related areas, we have created a unique large-scale dataset with various levels of human annotation combined with automatically recorded pro- ductivity features. The data comprises 20,000–45,000 sentences of industry-generated content for English from or into three morphologically rich languages and was collected as part of the EU-funded QT21 project. The PE of all four language pairs was performed using a tool to …

Content truncated to 500 words