Incorporating Copying Mechanism in Sequence-to-Sequence Learning

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1631–1640, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Incorporating Copying Mechanism in Sequence-to-Sequence Learning Jiatao Gu† Zhengdong Lu‡ Hang Li‡ Victor O.K. Li† †Department of Electrical and Electronic Engineering, The University of Hong Kong {jiataogu, vli}@eee.hku.hk ‡Huawei Noah’s Ark Lab, Hong Kong {lu.zhengdong, hangli.hl}@huawei.com Abstract We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which cer- tain segments in the input sequence are selectively replicated in the output se- quence. A similar phenomenon is ob- servable in human language communica- tion. For example, humans tend to re- peat entity names or even long phrases in conversation. The challenge with re- gard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network- based Seq2Seq learning and propose a new model called COPYNET with encoder- decoder structure. COPYNET can nicely integrate the regular way of word gener- ation in the decoder with the new copy- ing mechanism which can choose sub- sequences in the input sequence and put them at proper places in the output se- quence. Our empirical study on both syn- thetic data sets and real world data sets demonstrates the efficacy of COPYNET. For example, COPYNET can outperform regular RNN-based model with remark- able margins on text summarization tasks. 1 Introduction Recently, neural network-based sequence-to- sequence learning (Seq2Seq) has achieved re- markable success in various natural language pro- cessing (NLP) tasks, including but not limited to Machine Translation (Cho et al., 2014; Bahdanau et al., 2014), Syntactic Parsing (Vinyals et al., 2015b), Text Summarization (Rush et al., 2015) and Dialogue Systems (Vinyals and Le, 2015). Seq2Seq is essentially an encoder-decoder model, in which the encoder first transform the input se- quence to a certain representation which can then transform the representation into the output se- quence. Adding the attention mechanism (Bah- danau et al., 2014) to Seq2Seq, first proposed for automatic alignment in machine translation, has led to significant improvement on the perfor- mance of various tasks (Shang et al., 2015; Rush et al., 2015). Different from the canonical encoder- decoder architecture, the attention-based Seq2Seq model revisits the input sequence in its raw form (array of word representations) and dynamically fetches the relevant piece of information based mostly on the feedback from the generation of the output sequence. In this paper, we explore another mechanism important to the human language communication, called the “copying mechanism”. Basically, it refers to the mechanism that locates a certain seg- ment of the input sentence and puts the segment into the output sequence. For example, in the following two dialogue turns we observe differ- ent patterns in which some subsequences (colored blue) in the response (R) are copied from the input utterance (I): I: Hello Jack, my name is Chandralekha. R: Nice to meet you, Chandralekha. I: This new guy doesn’t perform exactly as we expected. …

Content truncated to 500 words