Deep Copycat Networks for Text-to-Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3227–3236, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3227 Deep copycat Networks for Text-to-Text Generation Julia Ive1, Pranava Madhyastha2 and Lucia Specia2 1DCS, University of Sheffield, UK 2Department of Computing, Imperial College London, UK j.ive@sheffield.ac.uk {pranava,l.specia}@imperial.ac.uk Abstract Most text-to-text generation tasks, for example text summarisation and text simplification, re- quire copying words from the input to the out- put. We introduce copycat, a transformer- based pointer network for such tasks which obtains competitive results in abstractive text summarisation and generates more abstractive summaries. We propose a further extension of this architecture for automatic post-editing, where generation is conditioned over two in- puts (source language and machine transla- tion), and the model is capable of decid- ing where to copy information from. This approach achieves competitive performance when compared to state-of-the-art automated post-editing systems. More importantly, we show that it addresses a well-known limita- tion of automatic post-editing – overcorrecting translations – and that our novel mechanism for copying source language words improves the results. 1 Introduction Text-to-text generation is generally addressed using sequence to sequence neural models (Sutskever et al., 2014). While initially pro- posed for machine translation (Bahdanau et al., 2015), these models have been shown to perform very well in monolingual variants of these tasks, such as text summarisation and text simplification (Nallapati et al., 2016a; Scarton and Specia, 2018). One limitation of such models is that they tend to be “over creative” when generating outputs, i.e. create a completely new output, which is not what they should do. Different strategies have been used to mitigate this issue, primarily in the con- text of abstractive text summarisation. The most prominent strategy is the use of pointer networks, which substantially outperform vanilla sequence to sequence models (Nallapati et al., 2016b; Gu et al., 2016; See et al., 2017). Pointer networks (Vinyals et al., 2015) are an extension of attentive recurrent neural network (RNN) architectures to use attention as a pointer to select which tokens of the input sequence should be copied to the output, as a switching mechanism between copying and generating new words. In this paper we propose copycat, a flexible pointer network framework for text-to-text gen- eration tasks. It differs from previous work in the following main ways: it is based on trans- former networks (Vaswani et al., 2017), which have been shown to achieve superior performance in text generation tasks. While Gehrmann et al. (2018b) also provide contrastive experiments us- ing transformers, they however randomly choose one of the attention heads as the copy-distribution, while we let the networks learn through all layers and use the attention heads from the final layer of the decoder to learn the distribution; thereby the decision between copying input words and gener- ating new words is made at the highest level of abstraction. Further, in our model, we do not use coverage penalty …

Content truncated to 500 words