Tools and Guidelines for Principled Machine Translation Development

Tools and Guidelines for Principled Machine Translation Development Nora Aranberri,∗ Eleftherios Avramidis,† Aljoscha Burchardt,† Ondˇrej Klejch,‡ Martin Popel‡ and Maja Popovi´c∇ ∗University of the Basque Country (UPV/EHU), IXA Group – nora.aranberri@ehu.eus †DFKI Berlin – {eleftherios.avramidis, aljoscha.burchardt}@dfki.de ‡Charles University in Prague, Faculty of Mathematics and Physics, UFAL – {klejch, popel}@ufal.mff.cuni.cz ∇Humboldt University of Berlin – maja.popovic@hu-berlin.de Abstract This work addresses the need to aid Machine Translation (MT) development cycles with a complete workflow of MT evaluation methods. Our aim is to assess, compare and improve MT system variants. We hereby report on novel tools and practices that support various measures, developed in order to support a principled and informed approach of MT development. Our toolkit for automatic evaluation showcases quick and detailed comparison of MT system variants through automatic metrics and n-gram feedback, along with manual evaluation via edit-distance, error annotation and task-based feedback. Keywords: evaluation methodologies; machine translation; tools, systems, applications 1. Introduction Machine Translation (MT) development cycles depend on suitable MT evaluation tools and methods that help to as- sess, compare, and improve system variants. Yet, MT eval- uation is a notoriously difficult topic due to the variability of the language that makes it impossible to anticipate one or more “correct” translations in advance. Therefore, a full MT development process often relies on a mix of automatic measures and human rating. Automatic measures are com- puted against human reference translations in order to set up baselines, building corpora, quick comparisons etc. Hu- man rating, such as comparison of MT output, is used at certain milestones for comparing the most prominent sys- tems. More recently, human post-editing and explicit error mark-up have been used to bridge the gap towards the use of MT in language industry, where the post-editing effort and the (absence of major) errors are decisive factors for deter- mining the usability of MT engines for a given task. In this work, we report on novel tools – one for automatic evalu- ation and one for human evaluation – and report on prac- tices that we have developed in the context of the QTLeap project1 in order to support a principled and informed ap- proach to MT development. The structure of this paper is as follows: Section 2. outlines the main functionality of MT-COMPAREVAL,2 an open- source tool for MT development that has been developed within the project and is used to “automatically” monitor MT improvements. Section 3. describes the manual evalu- ation being performed in the project on a regular basis for gaining insights based on qualitative inspection. Finally, Section 4. concludes this paper. 2. Developer’s Workbench: MT-COMPAREVAL MT-COMPAREVAL is an open-source tool that has been designed to help MT developers to compare and evaluate various MT engines/experiments and settings using several 1http://qtleap.eu 2https://github.com/choko/MT-ComparEval measures that represent the current best practices. The sys- tem helps to tie the development cycles together by linking three ingredients: • An evaluation panel that provides a graphical inter- face for comparing the performance of various sys- tems on the same output, …

Content truncated to 500 words