Encode, Tag, Realize: High-Precision Text Editing

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5054–5065, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5054 Encode, Tag, Realize: High-Precision Text Editing Eric Malmi Google Research emalmi@google.com Sebastian Krause Google Research bastik@google.com Sascha Rothe Google Research rothe@google.com Daniil Mirylenka Google Research dmirylenka@google.com Aliaksei Severyn Google Research severyn@google.com Abstract We propose LASERTAGGER—a sequence tag- ging approach that casts text generation as a text editing task. Target texts are recon- structed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To pre- dict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, ab- stractive summarization, and grammar correc- tion. LASERTAGGER achieves new state-of- the-art results on three of these tasks, performs comparably to a set of strong seq2seq base- lines with a large number of training exam- ples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than compara- ble seq2seq models, making it more attractive for running in a live environment. 1 Introduction Neural sequence-to-sequence (seq2seq) models provide a powerful framework for learning to trans- late source texts into target texts. Since their first application to machine translation (MT) (Sutskever et al., 2014) they have become the de facto ap- proach for virtually every text generation task, in- cluding summarization (Tan et al., 2017), image captioning (Xu et al., 2015), text style transfer (Rao and Tetreault, 2018; Nikolov and Hahnloser, 2018; Jin et al., 2019), and grammatical error correction (Chollampatt and Ng, 2018; Grundkiewicz et al., 2019). We observe that in some text generation tasks, such as the recently introduced sentence splitting and sentence fusion tasks, output texts highly over- lap with inputs. In this setting, learning a seq2seq model to generate the output text from scratch Encode Tag Turing was born in 1912 . Turing died in 1954 . Turing was born in 1912 and he died in 1954 . Realize KEEP KEEP KEEP KEEP KEEP and heDELETE DELETE KEEP KEEP KEEP KEEP Figure 1: LASERTAGGER applied to sentence fusion. seems intuitively wasteful. Copy mechanisms (Gu et al., 2016; See et al., 2017) allow for choosing between copying source tokens and generating ar- bitrary tokens, but although such hybrid models help with out-of-vocabulary words, they still re- quire large training sets as they depend on output vocabularies as large as those used by the standard seq2seq approaches. In contrast, we propose learning a text editing model that applies a set of edit operations on the input sequence to reconstruct the output. We show that it is often enough to use a relatively small set of output tags representing text deletion, rephrasing and word reordering to be able to …

Content truncated to 500 words