A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3050–3056 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3050 A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning Gonc¸alo M. Correia Instituto de Telecomunicac¸˜oes Lisbon, Portugal goncalo.correia@lx.it.pt Andr´e F. T. Martins Instituto de Telecomunicac¸˜oes & Unbabel Lisbon, Portugal andre.martins@unbabel.com Abstract Automatic post-editing (APE) seeks to auto- matically refine the output of a black-box ma- chine translation (MT) system through human post-edits. APE systems are usually trained by complementing human post-edited data with large, artificial data generated through back- translations, a time-consuming process often no easier than training a MT system from scratch. In this paper, we propose an alter- native where we fine-tune pre-trained BERT models on both the encoder and decoder of an APE system, exploring several parameter shar- ing strategies. By only training on a dataset of 23K sentences for 3 hours on a single GPU we obtain results that are competitive with sys- tems that were trained on 5M artificial sen- tences. When we add this artificial data, our method obtains state-of-the-art results. 1 Introduction The goal of automatic post-editing (APE; Simard et al., 2007) is to automatically correct the mistakes produced by a black-box machine translation (MT) system. APE is particularly appealing for rapidly customizing MT, avoiding to train new systems from scratch. Interfaces where human translators can post-edit and improve the quality of MT sen- tences (Alabau et al., 2014; Federico et al., 2014; Denkowski, 2015; Hokamp, 2018) are a common data source for APE models, since they provide triplets of source sentences (src), machine trans- lation outputs (mt), and human post-edits (pe). Unfortunately, human post-edits are typically scarce. Existing APE systems circumvent this by generating artificial triplets (Junczys-Dowmunt and Grundkiewicz, 2016; Negri et al., 2018). How- ever, this requires access to a high quality MT sys- tem, similar to (or better than) the one used in the black-box MT itself. This spoils the motivation of APE as an alternative to large-scale MT training in the first place: the time to train MT systems in order to extract these artificial triplets, combined with the time to train an APE system on the result- ing large dataset, may well exceed the time to train a MT system from scratch. Meanwhile, there have been many successes of transfer learning for NLP: models such as CoVe (McCann et al., 2017), ELMo (Peters et al., 2018), OpenAI GPT (Radford et al., 2018), ULM- FiT (Howard and Ruder, 2018), and BERT (Devlin et al., 2019) obtain powerful representations by training large-scale language models and use them to improve performance in many sentence-level and word-level tasks. However, a language generation task such as APE presents additional challenges. In this paper, we build upon the successes above and show that transfer learning is an effective and time-efficient strategy for APE, using a pre- trained BERT model. This is an appealing strategy in practice: while large language models like BERT are expensive …

Content truncated to 500 words