Seq2Edits: Sequence Transduction Using Span-level Edit Operations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5147–5159, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5147 Seq2Edits: Sequence Transduction Using Span-level Edit Operations Felix Stahlberg and Shankar Kumar Google Research {fstahlberg,shankarkumar}@google.com Abstract We propose Seq2Edits, an open-vocabulary approach to sequence editing for natural lan- guage processing (NLP) tasks with a high de- gree of overlap between input and output texts. In this approach, each sequence-to-sequence transduction is represented as a sequence of edit operations, where each operation either replaces an entire source span with target to- kens or keeps it unchanged. We evaluate our method on five NLP tasks (text normalization, sentence fusion, sentence splitting & rephras- ing, text simplification, and grammatical er- ror correction) and report competitive results across the board. For grammatical error cor- rection, our method speeds up inference by up to 5.2x compared to full sequence models be- cause inference time depends on the number of edits rather than the number of target tokens. For text normalization, sentence fusion, and grammatical error correction, our approach im- proves explainability by associating each edit operation with a human-readable tag. 1 Introduction Neural models that generate a target sequence con- ditioned on a source sequence were initially pro- posed for machine translation (MT) (Sutskever et al., 2014; Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2015; Vaswani et al., 2017), but are now used widely as a central component of a variety of NLP systems (e.g. Tan et al. (2017); Chollampatt and Ng (2018)). Raffel et al. (2019) argue that even problems that are traditionally not viewed from a sequence transduction perspective can benefit from massive pre-training when framed as a text-to-text problem. However, for many NLP tasks such as correcting grammatical errors in a sentence, the input and output sequence may over- lap significantly. Employing a full sequence model in these cases is often wasteful as most tokens are simply copied over from the input to the output. Another disadvantage of a full sequence model is that it does not provide an explanation for why it proposes a particular target sequence. In this work, inspired by a recent increased in- terest in text-editing (Dong et al., 2019; Malmi et al., 2019; Mallinson et al., 2020; Awasthi et al., 2019), we propose Seq2Edits, a sequence editing model which is tailored towards problems that re- quire only small changes to the input. Rather than generating the target sentence as a series of tokens, our model predicts a sequence of edit operations that, when applied to the source sentence, yields the target sentence. Each edit operates on a span in the source sentence and either copies, deletes, or replaces it with one or more target tokens. Edits are generated auto-regressively from left to right using a modified Transformer (Vaswani et al., 2017) ar- chitecture to facilitate learning of long-range depen- dencies. We apply our edit operation based model to five NLP tasks: text normalization, sentence fusion, sentence splitting & rephrasing, text simpli- fication, and grammatical …

Content truncated to 500 words