Parallel Iterative Edit Models for Local Sequence Transduction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4260–4270, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4260 Parallel Iterative Edit Models for Local Sequence Transduction Abhijeet Awasthi∗, Sunita Sarawagi , Rasna Goyal , Sabyasachi Ghosh , Vihari Piratla Department of Computer Science and Engineering, IIT Bombay Abstract We present a Parallel Iterative Edit (PIE) model for the problem of local sequence trans- duction arising in tasks like Grammatical er- ror correction (GEC). Recent approaches are based on the popular encoder-decoder (ED) model for sequence to sequence learning. The ED model auto-regressively captures full de- pendency among output tokens but is slow due to sequential decoding. The PIE model does parallel decoding, giving up the advan- tage of modelling full dependency in the out- put, yet it achieves accuracy competitive with the ED model for four reasons: 1. predict- ing edits instead of tokens, 2. labeling se- quences instead of generating sequences, 3. it- eratively refining predictions to capture de- pendencies, and 4. factorizing logits over ed- its and their token argument to harness pre- trained language models like BERT. Experi- ments on tasks spanning GEC, OCR correc- tion and spell correction demonstrate that the PIE model is an accurate and significantly faster alternative for local sequence transduc- tion. The code and pre-trained models for GEC are available at https://github. com/awasthiabhijeet/PIE. 1 Introduction In local sequence transduction (LST) an input se- quence x1, . . . , xn needs to be mapped to an out- put sequence y1, . . . , ym where the x and y se- quences differ only in a few positions, m is close to n, and xi, yj come from the same vocabulary Σ. An important application of local sequence trans- duction that we focus on in this paper is Gram- matical error correction (GEC). We contrast lo- cal transduction with more general sequence trans- duction tasks like translation and paraphrasing which might entail different input-output vocab- ulary and non-local alignments. The general se- ∗Correspondence to: awasthi@cse.iitb.ac.in quence transduction task is cast as sequence to sequence (seq2seq) learning and modeled popu- larly using an attentional encoder-decoder (ED) model. The ED model auto-regressively produces each token yt in the output sequence conditioned on all previous tokens y1, . . . , yt−1. Owing to the remarkable success of this model in challeng- ing tasks like translation, almost all state-of-the-art neural models for GEC use it (Zhao et al., 2019; Lichtarge et al., 2019; Ge et al., 2018b; Chol- lampatt and Ng, 2018b; Junczys-Dowmunt et al., 2018). We take a fresh look at local sequence trans- duction tasks and present a new parallel-iterative- edit (PIE) architecture. Unlike the prevalent ED model that is constrained to sequentially generat- ing the tokens in the output, the PIE model gener- ates the output in parallel, thereby substantially re- ducing the latency of sequential decoding on long inputs. However, matching the accuracy …

Content truncated to 500 words