A User Study of the Incremental Learning in NMT

A User Study of the Incremental Learning in NMT Miguel Domingo1 and Mercedes García-Martínez2 and Álvaro Peris3 and Alexandre Helle2 and Amando Estela2 and Laurent Bié2 and Francisco Casacuberta1 and Manuel Herranz2 1PRHLT Research Center - Universitat Politècnica de València {midobal, fcn}@prhlt.upv.es 2Pangeanic / B.I Europa - PangeaMT Technologies Division {m.garcia, a.helle, a.estela, l.bie, m.herranz}@pangeanic.com 3Independent Researcher lvapeab@gmail.com Abstract In the translation industry, human experts usu- ally supervise and post-edit machine trans- lation hypotheses. Adaptive neural machine translation systems, able to incrementally up- date the underlying models under an online learning regime, have been proven to be use- ful to improve the efficiency of this workflow. However, this incremental adaptation is some- what unstable, and it may lead to undesirable side effects. One of them is the sporadic ap- pearance of made-up words, as a byproduct of an erroneous application of subword segmen- tation techniques. In this work, we extend pre- vious studies on on-the-fly adaptation of neu- ral machine translation systems. We perform a user study involving professional, experienced post-editors, delving deeper on the aforemen- tioned problems. Results show that adaptive systems were able to learn how to generate the correct translation for task-specific terms, re- sulting in an improvement of the user’s produc- tivity. We also observed a close similitude, in terms of morphology, between made-up words and the words that were expected. 1 Introduction Despite its improvements and obtaining admissible re- sults in many tasks, machine translation (MT) is still very far from obtaining automatic high-quality trans- lations (Dale, 2016; Toral et al., 2018). Thus, a hu- man agent needs to supervise and correct the outputs generated by an MT system. This process is known as post-editing and is a common use case of MT in the in- dustrial environment. As MT systems are continuously improving their capabilities, it has acquired major rele- vance in the translation market (Guerberof, 2008; Pym et al., 2012; Hu and Cadwell, 2016; Turovsky, 2016). Throughout the post-editing process, new data are continuously generated. These new data have valuable properties—they are domain-specific training samples. Thus, it can be leveraged to continuously adapt the sys- c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. tem towards a given domain or the style of the post- editor. A common way of achieving this consists in following an online-learning paradigm (Ortiz-Martínez, 2016; Peris and Casacuberta, 2019). Each time the user validates a post-edit, the system’s models are up- dated incrementally with this new sample. Hence, when the system generates the next translation, it will con- sider the previous post-edits made by the user and it is expected to produce higher quality translations (or, at least, more suited to the post-editor’s preferences). Domingo et al. (2019b) conducted a preliminary user study for professional post-editors, who had a positive perception of the adaptive systems. However, they no- ticed that, in some cases, there were occurrences of some made-up words. In this work, we study the impact …

Content truncated to 500 words