An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 120–129, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing Marcin Junczys-Dowmunt Department for Natural Language Processing Adam Mickiewicz University in Pozna´n junczys@amu.edu.pl Roman Grundkiewicz School of Informatics University of Edinburgh rgrundki@inf.ed.ac.uk Abstract In this work, we explore multiple neu- ral architectures adapted for the task of automatic post-editing of machine trans- lation output. We focus on neural end- to-end models that combine both inputs mt (raw MT output) and src (source lan- guage input) in a single neural architec- ture, modeling {mt, src} → pe directly. Apart from that, we investigate the influ- ence of hard-attention models which seem to be well-suited for monolingual tasks, as well as combinations of both ideas. We report results on data sets provided dur- ing the WMT-2016 shared task on au- tomatic post-editing and can demonstrate that dual-attention models that incorporate all available data in the APE scenario in a single model improve on the best shared task system and on all other published re- sults after the shared task. Dual-attention models that are combined with hard atten- tion remain competitive despite applying fewer changes to the input. 1 Introduction Given the raw output of a (possibly unknown) ma- chine translation system from language src to lan- guage mt, Automatic Post-Editing (APE) is the process of automatic correction of raw MT output (mt), so that a closer resemblance to human post- edited MT output (pe) is achieved. While APE systems that only model mt → pe yield good re- sults, the field has always strived towards methods that also integrate src in various forms. With neural encoder-decoder models, and multi-source models in particular, this can be now achieved in more natural ways than for previously popular phrase-based statistical machine transla- tion (PB-SMT) systems. Despite this, previously reported results for multi-source or dual-source models in APE scenarios are unsatisfying in terms of performance. In this work, we explore a number of single- source and dual-source neural architectures which we believe to be better fits to the APE task than vanilla encoder-decoder models with soft atten- tion. We focus on neural end-to-end models that combine both inputs mt and src in a single neu- ral architecture, modeling {mt, src} → pe di- rectly. Apart from that, we investigate the influ- ence of hard-attention models, which seem to be well-suited for monolingual tasks. Finally, we cre- ate combinations of both architectures. We report results on data sets provided dur- ing the WMT-2016 shared task on automatic post- editing (Bojar et al., 2016) and compare our per- formance against the shared task winner, the sys- tem submitted by the Adam Mickiewicz Univer- sity (AMU) team (Junczys-Dowmunt and Grund- kiewicz, 2016), and a more recent system by Pal et al. (2017) with the previously best published re- sults on the same test set. Our main contributions are: (1) we perform a thorough comparison of end-to-end …

Content truncated to 500 words