FELIX: Flexible Text Editing Through Tagging and Insertion

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244–1255 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1244 FELIX: Flexible Text Editing Through Tagging and Insertion Jonathan Mallinson∗ Google Research Aliaksei Severyn∗ Google Research {jonmall,severyn,emalmi,ggarrido}@google.com Eric Malmi Google Research Guillermo Garrido Google Research Abstract We present FELIX – a flexible text-editing ap- proach for generation, designed to derive max- imum benefit from the ideas of decoding with bi-directional contexts and self-supervised pre- training. In contrast to conventional sequence- to-sequence (seq2seq) models, FELIX is effi- cient in low-resource settings and fast at in- ference time, while being capable of model- ing flexible input-output transformations. We achieve this by decomposing the text-editing task into two sub-tasks: tagging to decide on the subset of input tokens and their or- der in the output text and insertion to in-fill the missing tokens in the output not present in the input. The tagging model employs a novel Pointer mechanism, while the insertion model is based on a Masked Language Model (MLM). Both of these models are chosen to be non-autoregressive to guarantee faster in- ference. FELIX performs favourably when compared to recent text-editing methods and strong seq2seq baselines when evaluated on four NLG tasks: Sentence Fusion, Machine Translation Automatic Post-Editing, Summa- rization, and Text Simplification. 1 Introduction The ideas of text in-filling coupled with self- supervised pre-training of deep Transformer net- works on large text corpora have dramatically changed the landscape in Natural Language Un- derstanding. BERT (Devlin et al., 2019) and its successive refinements RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019) implement this recipe and have significantly pushed the state-of-the-art on multiple NLU benchmarks such as GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2016). More recently, masked or in-filling style objectives for model pretraining have been applied to seq2seq tasks, significantly pushing the state-of-the-art ∗Equal contribution. tagging KEEP KEEPINS KEEP DEL KEEP The very big [REPL] loud [/REPL] MASK cat TAGGER The big very loud cat POINTER insertion MASKED LANGUAGE MODEL The very big [REPL] loud [/REPL] old cat Figure 1: FELIX transforms the source “The big very loud cat” into the target text “The very big old cat”. on a number of text generation tasks, e.g, KER- MIT (Chan et al., 2019), MASS (Song et al., 2019), Bert2Bert (Rothe et al., 2020), BART (Lewis et al., 2020) and T5 (Raffel et al., 2019). While seq2seq frameworks offer a generic tool for modeling almost any kind of text-to-text trans- duction, there are still many real-world tasks where generating target texts completely from scratch— as is done with seq2seq approaches—can be un- necessary. This is especially true for monolingual settings where input and output texts have relatively high degrees of overlap. In such cases a natural approach is to cast conditional text generation as a text-editing task, where the model learns to recon- struct target texts by applying a set of edit opera- tions to the inputs. Typically, the set of edit opera- tions is …

Content truncated to 500 words