Recurrent Inference in Text Editing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1758–1769 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1758 Recurrent Inference in Text Editing Ning Shi Georgia Institute of Technology ning.shi@gatech.edu Ziheng Zeng University of Illinois zzeng13@illinois.edu Haotian Zhang Learnable, Inc. haotian.zhang@learnable.ai Yichen Gong Horizon Robotics, Inc. yichen01.gong@horizon.ai Abstract In neural text editing, prevalent sequence-to- sequence based approaches directly map the unedited text either to the edited text or the editing operations, in which the performance is degraded by the limited source text encod- ing and long, varying decoding steps. To ad- dress this problem, we propose a new infer- ence method, Recurrence, that iteratively per- forms editing actions, significantly narrowing the problem space. In each iteration, encoding the partially edited text, Recurrence decodes the latent representation, generates an action of short, fixed-length, and applies the action to complete a single edit. For a comprehensive comparison, we introduce three types of text editing tasks: Arithmetic Operators Restora- tion (AOR), Arithmetic Equation Simplifica- tion (AES), Arithmetic Equation Correction (AEC). Extensive experiments on these tasks with varying difficulties demonstrate that Re- currence achieves improvements over conven- tional inference methods. 1 Introduction For text editing, the sequence-to-sequence (seq2seq) framework has been applied to text simplification (Narayan and Gardent, 2014; Dong et al., 2019), punctuation restoration (Tilk and Alum¨ae, 2016; Kim, 2019), grammatical error correction (Ge et al., 2018; Lichtarge et al., 2018; Zhao et al., 2019), machine translation post-editing (Libovick´y et al., 2016; B´erard et al., 2017), and etc. We observe that current inference methods can be roughly grouped into two categories: End-to-end (End2end) (Nisioi et al., 2017; See et al., 2017; Tan et al., 2017; Junczys-Dowmunt et al., 2018) and Tagging (Filippova et al., 2015; Che et al., 2016; Libovick´y et al., 2016; Wang et al., 2017; Alva-Manchego et al., 2017; Kim, 2019). For models from both categories, the encoders extract and encode information from the source text sequence. Yet, the goal of the Figure 1: High-level illustration of End2end, Tagging, and Recurrence in text editing. decoders is different for End2end and Tagging. Upon receiving the encoder’s hidden states that comprise the source text information, the decoder of End2end directly decodes the hidden states and generates the completely edited target text sequence. But, the decoder of Tagging produces a sequence of editing operations, such as deletion and insertion, that is later applied to the source text to yield the edited text via a realization step (Malmi et al., 2019). The mechanisms of End2end and Tagging are illustrated in Figure 1. However, both End2end and Tagging are prob- lematic because as decoding progresses, the diver- gence between the partially edited text and the orig- inal text grows, rendering the encoder hidden states less and less helpful for decoding the edited text or editing operations toward the end of the editing pro- cess; and as the number of decoding steps increases with edited text length, decoding the completely edited text or the full editing operation sequence becomes more and more demanding. …

Content truncated to 500 words