Multidimensional assessment of the eTranslation output for English–Slovene

Multidimensional assessment of the eTranslation output for English–Slovene Mateja Arnejšek European Commission Directorate-General for Translation Slovene Language Department mateja.arnejsek@ec.europa.eu Alenka Unk European Commission Directorate-General for Translation Slovene Language Department alenka.unk@ec.europa.eu Abstract The Slovene language department of the European Commission Directorate- General for Translation has always been an early adopter of new developments in the area of machine translation. In 2018, the department started using neural ma- chine translation produced by the eTrans- lation in-house engines. In 2019, a multi- dimensional assessment of the eTransla- tion output for the language combination English–Slovene was carried out. It was based on two user satisfaction surveys, an analysis of reported errors and an ex post analysis of a sample. As part of the as- sessment effort, a categorisation of errors was devised in order to raise awareness among translators of the potential pitfalls of neural machine translation. 1 Machine translation in DGT eTranslation1 is one of the Building Blocks for a Digital Connected Europe in the framework of the Connecting Europe Facility (CEF).2 It was launched in November 2017 with the progressive addition of engines for different language combinations. eTranslation took over from MT@EC, which had been fully operational since June 2013. MT@EC was a statistical machine translation (SMT) system based on the MOSES open-source translation toolkit.3 The Directorate- General for Translation (DGT) of the European © 2020 Arnejšek, Mateja and Alenka Unk. This article is licensed under a Creative Commons 3.0 licence, no 1 derivative works, attribution, CC-BY-ND. https://webgate.ec.europa.eu/etranslation. 2 CEF is a key EU funding instrument to promote growth, jobs and competitiveness through targeted infra- 3 structure investment at European level. Koehn et al. (2007). Commission had developed MT@EC under the Interoperability Solutions for European Public Administrations (ISA) programme with co- funding from EU research and innovation programmes. CEF eTranslation followed the field’s move into neural machine translation (NMT). DGT is organised into language departments (LDs), one for each official language of the EU.4 Right from the launch of the NMT engines, LDs were provided with practical guidelines that aim to ensure that machine translation is used con- sistently and effectively within DGT, encourag- ing translators to at the very least try using ma- chine translation, but still allowing for different approaches to cater to specific needs. Training has also been organised to present the new tech- nology and its known general pitfalls. Based on the guidelines and the training, the LDs adopted different approaches to the uptake of NMT and used it in different ways and to differing extents. In autumn 2018, after the initial period of in- troduction, uptake and testing, DGT decided to assess NMT output in the LDs, gathering general opinions on how useful neural engines are for the individual LDs and on the kind of impact these engines can have on the efficiency and quality of translation. The objectives of the exercise were to check which of the two engines, NMT or SMT, was preferred as the default engine in the automated pre-processing of translation requests and …

Content truncated to 500 words