Evaluating Domain Adaptation for Machine Translation Across Scenarios

Evaluating Domain Adaptation for Machine Translation Across Scenarios Thierry Etchegoyhen1, Anna Fern´andez Torn´e2, Andoni Azpeitia1, Eva Mart´ınez Garcia1 and Anna Matamala2 1Vicomtech, Donostia / San Sebasti´an, Spain 2Universitat Aut`onoma de Barcelona, Barcelona, Spain 1{tetchegoyhen, aazpeitia, emartinez}@vicomtech.org 2{Ana.Fernandez.Torne, Anna.Matamala}@uab.cat Abstract We present an evaluation of the benefits of domain adaptation for machine translation, on three separate domains and language pairs, with varying degrees of domain specificity and amounts of available training data. Domain-adapted statistical and neural machine translation systems are compared to each other and to generic online systems, thus providing an evaluation of the main options in terms of machine translation. Alongside automated translation metrics, we present experimental results involving professional translators, in terms of quality assessment, subjective evaluations of the task and post-editing productivity measurements. The results we present quantify the clear advantages of domain adaptation for machine translation, with marked impacts for domains with higher specificity. Additionally, the results of the experiments show domain-adapted neural machine translation systems to be the optimal choice overall. Keywords: Machine Translation, Domain Adaptation, Quality Evaluation, Productivity Evaluation, Subjective Evaluation 1. Introduction Statistical machine translation (SMT) (Brown et al., 1990) has been the dominant approach to automated translation for the last two decades, with neural machine translation (NMT) (Bahdanau et al., 2015) quickly becoming the new main paradigm in academic research and the industry, on the basis of the improvements it provides across the board (Toral and S´anchez-Cartagena, 2017). The data-driven na- ture of both approaches conditions the quality of their out- put to the availability of large volumes of adequate training resources for a given domain. However, domain-specific resources are usually scarce, thus making proper domain adaptation as much a challenge as it is a goal in developing accurate machine translation (MT) systems. Domain adaptation has been extensively explored within SMT, with numerous studies focusing on the selection of supplementary data (Axelrod et al., 2011; Gasc´o et al., 2012; Eetemadi et al., 2015), translation model combina- tion (Foster and Kuhn, 2007; Sennrich, 2012) or the inte- gration of external information (Bisazza et al., 2011), to cite only a few. In NMT, domain adaptation is a more recent endeavour, with fine tuning currently the main method to gear generic translation networks towards specific domains (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016; Crego et al., 2016). Progress in machine translation technology has also given rise to large generic machine translation systems, many of which are freely available online. The increasing trans- lation quality they provide, in part due to growing user feedback and training data covering multiple domains, has made them popular alternatives even in cases where domain-adapted systems might be better suited, although this specific aspect has not been fully evaluated yet. So far, large online translation systems have been essentially compared to academic systems for news-related translation (see, e.g., (Toral et al., 2011; Bojar et al., 2016)), rather than to systems tuned for the kind of specific domains that are more typical in the translation industry. Machine translation quality and usefulness …

Content truncated to 500 words