Evaluating the usefulness of neural machine translation for the Polish translators in the European Commission

Evaluating the usefulness of neural machine translation for the Polish translators in the European Commission Karolina Stefaniak European Commission, Directorate General for Translation 1-7 Boulevard Pierre Frieden, L-1543 Luxembourg Office: T2 00/B076 karolina.stefaniak@ec.europa.eu Abstract The mission of the Directorate General for Translation (DGT) is to provide high- quality translation to help the European Commission communicate with EU citi- zens. To this end DGT employs almost 2000 translators from all EU official lan- guages. But while the demand for transla- tion has been continuously growing, fol- lowing a global trend, the number of translators has decreased. To cope with the demand, DGT extensively uses a CAT environment encompassing transla- tion memories, terminology databases and recently also machine translation. This paper examines the benefits and risks of using neural machine translation to augment the productivity of in‒house DGT translators for the English‒Polish language pair. Based on the analysis of a sample of NMT-translated texts and on the observations of the working practices of Polish translators it is concluded that the possible productivity gain is still modest, while the risks to quality are quite substantial. 1 Introduction Machine translation arrived at the European Commission in 1976 with the purchase of Systran, a rule-based technology, for the English- French language pair. This initial version was then developed and adapted as EC Systran to respond to the Commission’s needs; specialized terminology and dictionaries were imported, and, over time, other language pairs were added. By © 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, at- tribution, CC-BY-ND. 2010, some 2 million pages were translated per year with Systran, used both by EU and Member States' officials to provide quick drafts of texts in unfamiliar languages, and by EU translators. However, while the quality was fine for getting the gist of short, repetitive texts with standardized structure and terminology, the system was rather unsuitable for translating legislation, and so its use among translators never really caught up (Petritis, 2001; Eisele, 2017a). EC Systran was discontinued in December 2010 and since then the Commission has been working on its own machine translation system, developed by the Directorate-General for Translation. MT@EC, based on Moses, an open- source statistical machine translation (SMT) toolkit, and improved by rule-based pre- and post-processing, went operational in June 2013. By 2017 it offered 78 direct language pairs, covering all EU official languages (Eisele 2017b). It proved to be quite helpful for certain language pairs (English‒French or English- Portuguese, for example) and quite unusable for other (like English-Hungarian or English- Finnish). Polish, with its free word order, rich inflectional morphology and complex orthography, was also quite challenging for the system, which produced very mixed results, from acceptable translations to unintelligible nonsense. As a result, few Polish translators tried to use it as a resource in the translation process. An internal evaluation performed in the Polish Language Department of DGT in 2017 revealed that SMT was useful only for certain text types and that only half of the …

Content truncated to 500 words