Keeping Notes: Conditional Natural Language Generation with a Scratchpad Encoder

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4157–4167 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4157 Keeping Notes: Conditional Natural Language Generation with a Scratchpad Mechanism Ryan Y. Benmalek†∗, Madian Khabsa‡∗, Suma Desu¶∗, Claire Cardie†, Michele Banko§∗ †Cornell University, ‡Facebook, ¶Independent , §Sentropy Technologies ryanai3@cs.cornell.edu, mkhabsa@fb.com, desuma24@gmail.com, mbanko@sentropy.io, cardie@cs.cornell.edu Abstract We introduce the Scratchpad Mechanism, a novel addition to the sequence-to-sequence (seq2seq) neural network architecture and demonstrate its effectiveness in improving the overall fluency of seq2seq models for natural language generation tasks. By enabling the decoder at each time step to write to all of the encoder output layers, Scratchpad can em- ploy the encoder as a “scratchpad” memory to keep track of what has been generated so far and thereby guide future generation. We evaluate Scratchpad in the context of three well-studied natural language generation tasks — Machine Translation, Question Generation, and Text Summarization — and obtain state- of-the-art or comparable performance on stan- dard datasets for each task. Qualitative as- sessments in the form of human judgements (question generation), attention visualization (MT), and sample output (summarization) pro- vide further evidence of the ability of Scratch- pad to generate fluent and expressive output. 1 Introduction The sequence-to-sequence neural network frame- work (seq2seq) (Sutskever et al., 2014) has been successful in a wide range of tasks in natural lan- guage processing, from machine translation (Bah- danau et al., 2014) and semantic parsing (Dong and Lapata, 2016) to summarization (Nallapati et al., 2016b; See et al., 2017). Despite this suc- cess, seq2seq models are known to often exhibit an overall lack of fluency in the natural language output produced: problems include lexical rep- etition, under-generation in the form of partial phrases and lack of specificity (often caused by the gap between the input and output vocabular- ies) (Xie, 2017). Recently, a number of task- specific attention variants have been proposed to deal with these issues: See et al. (2017) intro- duced a coverage mechanism (Tu et al., 2016) ∗ Work performed while at Apple. to deal with repetition and over-copying in sum- marization, Hua and Wang (2018) introduced a method of attending over keyphrases to improve argument generation, and Kiddon et al. (2016) introduced a method that attends to an agenda of items to improve recipe generation. Perhaps not surprisingly, general-purpose attention mech- anisms targeting individual problems from the list above have also begun to be developed. Copy- net (Gu et al., 2016) and pointer-generator net- works (Vinyals et al., 2015), for example, aim to reduce input-output vocabulary mismatch and, thereby, improve specificity, while the coverage- based techniques of Tu et al. (2016) tackle rep- etition and under-generation. These techniques, however, often require significant hyperparameter tuning and are purposely limited to fixing a spe- cific problem in the generated text. We present here a general-purpose addition to the standard seq2seq framework that aims to si- multaneously tackle all of the above issues. In particular, we propose Scratchpad, a novel …

Content truncated to 500 words