Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings

Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings Ant´onio G´ois* Unbabel Lisbon, Portugal antoniogois@gmail.com Kyunghyun Cho New York University Facebook AI New York, USA kyunghyun.cho@nyu.edu Andr´e Martins Unbabel Instituto de Telecomunicac¸˜oes Lisbon, Portugal andre.martins@unbabel.com Abstract Recent research in neural machine trans- lation has explored flexible generation or- ders, as an alternative to left-to-right gen- eration. However, training non-monotonic models brings a new complication: how to search for a good ordering when there is a combinatorial explosion of orderings ar- riving at the same final result? Also, how do these automatic orderings compare with the actual behaviour of human translators? Current models rely on manually built bi- ases or are left to explore all possibilities on their own. In this paper, we analyze the orderings produced by human post-editors and use them to train an automatic post- editing system. We compare the resulting system with those trained with left-to-right and random post-editing orderings. We ob- serve that humans tend to follow a nearly left-to-right order, but with interesting de- viations, such as preferring to start by cor- recting punctuation or verbs. 1 Introduction Neural sequence generation models have been widely adopted for tasks such as machine trans- lation (MT) (Bahdanau et al., 2015; Vaswani et al., 2017) and automatic post-editing of transla- tions (Junczys-Dowmunt and Grundkiewicz, 2016; Chatterjee et al., 2016; Correia and Martins, 2019; Lopes et al., 2019). These models typically gen- erate one word at a time, and rely on a factoriza- c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. *Work partly done during a research visit at New York Uni- versity. 0: Die LMS ge¨offnet ist . [ I:2:ist ] 1: Die LMS ist ge¨offnet ist . [ D:4:ist ] 2: Die LMS ist ge¨offnet . Table 1: Example of a small post-edit from the training set. Each action is represented by three features: its type (I for insert and D for delete), its position in the sentence and the token to insert/delete. In this example, the token marked in red needs to be removed since it is incorrectly placed. The blue token is inserted to obtain the correct pe. tion that imposes a left-to-right generation order- ing. Recent alternatives allow for different gener- ation orderings (Welleck et al., 2019; Stern et al., 2019; Gu et al., 2019a), or even for parallel gen- eration of multiple tokens (Gu et al., 2018; Stern et al., 2019; Gu et al., 2019b; Zhou et al., 2020), which allows exploiting dependencies among non- consecutive tokens. One potential difficulty when training non-monotonic models is how to learn a good generation ordering. There are exponen- tially many valid orderings to generate a given sequence, and a model should prefer those that lead to accurate translations and can be efficiently learned. In previous work, to guide the search for a good ordering, oracle policies have been provided (Welleck et al., 2019), or another kind of inductive bias such as a loss function …

Content truncated to 500 words