A human evaluation of English-Irish statistical and neural machine translation

A human evaluation of English-Irish statistical and neural machine translation Meghan Dowling meghan.dowling@adaptcentre.ie Joss Moorkens joss.moorkens@adaptcentre.ie Andy Way andy.way@adaptcentre.ie ADAPT Centre, Dublin City University, Dublin, Ireland Sheila Castilho sheila.castilho@adaptcentre.ie Teresa Lynn teresa.lynn@adaptcentre.ie Abstract With official status in both Ireland and the EU, there is a need for high-quality English-Irish (EN-GA) machine transla- tion (MT) systems which are suitable for use in a professional translation environ- ment. While we have seen recent research on improving both statistical MT and neu- ral MT for the EN-GA pair, the results of such systems have always been reported using automatic evaluation metrics. This paper provides the first human evaluation study of EN-GA MT using professional translators and in-domain (public adminis- tration) data for a more accurate depiction of the translation quality available via MT. 1 Introduction The Irish language enjoys the status of both the first official language of Ireland and an official Eu- ropean Union language. As a result of this sta- tus is there is a requirement for official public con- tent to be made available in Irish in both Ireland1 and the EU.2 There is currently a derogation on the amount of Irish content published by the EU, due to be lifted at the end of 2021 (Publications Office of the European Union, 2011). At this point, the already high demand for professional Irish transla- tors will increase significantly. With this demand for the production of Irish-language text, usually c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1The Official Languages Act (2003) requires all official pub- lic information and services to be available in both Irish and English: http://www.irishstatutebook.ie/ eli/2003/act/32/enacted/en/html 2Irish has been a full EU language since 2006. with English as the source language, it is impor- tant that any available EN→GA MT systems are robust and fit-for-purpose. Despite MT having been established as a useful tool in the workflow of a professional translator, it is not yet the norm for Irish translators, whether freelance or within a translation company.3 As a lesser-resourced and minority language, Irish faces a barrier to state-of-the-art technology shown to be effective for majority languages (European Lan- guage Resource Coordination, 2020). While there has been research on improving EN→GA MT (Dowling et al., 2015; Arcan et al., 2016; Defauw et al., 2019; Dowling et al., 2019) to date there have been no publications describing a human evaluation (HE) study for EN→GA MT. This study aims to provide the first EN→GA MT HE study, investigating the measurable usefulness of EN→GA in a professional translation capacity. In an attempt to closely match the context in which EN→GA MT is intended to be used, professional translators will undertake post-editing (PE) tasks using MT output. Another aim of this study is to provide a human- derived comparison of EN→GA statistical ma- chine translation (SMT) and neural machine trans- lation (NMT). In previous work, a preliminary comparison of EN→GA SMT and NMT showed that SMT fared better than …

Content truncated to 500 words