Text Generation with Exemplar-based Adaptive Decoding

Proceedings of NAACL-HLT 2019, pages 2555–2565 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2555 Text Generation with Exemplar-based Adaptive Decoding Hao Peng♠⇤ Ankur P. Parikh} Manaal Faruqui} Bhuwan Dhingra|⇤ Dipanjan Das} ♠ Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA } Google AI Language, New York, NY | School of Computer Science, Carnegie Mellon University, Pittsburgh, PA hapeng@cs.washington.edu, {aparikh,mfaruqui,dipanjand}@google.com, bdhingra@andrew.cmu.edu Abstract We propose a novel conditioned text gener- ation model. It draws inspiration from tra- ditional template-based text generation tech- niques, where the source provides the con- tent (i.e., what to say), and the template influ- ences how to say it. Building on the success- ful encoder-decoder paradigm, it first encodes the content representation from the given in- put text; to produce the output, it retrieves exemplar text from the training data as “soft templates,” which are then used to construct an exemplar-specific decoder. We evaluate the proposed model on abstractive text sum- marization and data-to-text generation. Em- pirical results show that this model achieves strong performance and outperforms compara- ble baselines. 1 Introduction Conditioned text generation is the essence of many natural language processing (NLP) tasks, e.g., text summarization (Mani, 1999), machine translation (Koehn, 2009), and data-to-text gener- ation (Kukich, 1983; McKeown, 1992; Reiter and Dale, 1997). In its common neural sequence-to- sequence formulation (Sutskever et al., 2014; Cho et al., 2014), an encoder-decoder architecture is used. The decoder generates the text autoregres- sively, token-by-token, conditioning on the feature representations encoded from the source, typically with attention (Bahdanau et al., 2015) and copy mechanisms (Gu et al., 2016; See et al., 2017). This paradigm is capable of generating fluent ab- stractive text, but in an uncontrolled and some- times unreliable way, often producing degenerate outputs and favoring generic utterances (Vinyals and Le, 2015; Li et al., 2016). The encoder-decoder approach differs con- siderably from earlier template-based meth- ⇤ Work done during internship at Google. ods (Becker, 2002; Foster and White, 2004; Re- iter et al., 2005; Gatt and Reiter, 2009, inter alia), where the source content is filled into the slots of a handcrafted template. These solutions of- fer higher generation precision compared to neu- ral approaches (Wiseman et al., 2017), but tend to lack the naturalness of neural systems, and are less scalable to open domain settings, where the number of required templates can be prohibitively large. To sidestep the scalability problems with hand- crafted templates, it has been proposed to use simi- lar training samples as exemplars, to guide the de- coding process (Gu et al., 2018; Guu et al., 2018; Weston et al., 2018; Pandey et al., 2018; Cao et al., 2018a, inter alia).1 In general, existing methods accomplish this by (a) using traditional informa- tion retrieval (IR) techniques for exemplar extrac- tion (e.g., TF-IDF), and then (b) concatenating the exemplar to the source as additional inputs, allow- ing the decoder to attend over and copy from both. We propose a different strategy for …

Content truncated to 500 words