Controlling Output Length in Neural Encoder-Decoders

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1328–1338, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Controlling Output Length in Neural Encoder-Decoders Yuta Kikuchi1∗ kikuchi@lr.pi.titech.ac.jp Graham Neubig2† gneubig@cs.cmu.edu Ryohei Sasano1 sasano@pi.titech.ac.jp Hiroya Takamura1 takamura@pi.titech.ac.jp Manabu Okumura1 oku@pi.titecjh.ac.jp 1Tokyo Institute of Technology, Japan 2Carnegie Mellon University, USA Abstract Neural encoder-decoder models have shown great success in many sequence generation tasks. However, previous work has not in- vestigated situations in which we would like to control the length of encoder-decoder out- puts. This capability is crucial for applica- tions such as text summarization, in which we have to generate concise summaries with a desired length. In this paper, we pro- pose methods for controlling the output se- quence length for neural encoder-decoder models: two decoding-based methods and two learning-based methods.1 Results show that our learning-based methods have the capabil- ity to control length without degrading sum- mary quality in a summarization task. 1 Introduction Since its first use for machine translation (Kalch- brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014), the encoder-decoder ap- proach has demonstrated great success in many other sequence generation tasks including image caption generation (Vinyals et al., 2015b; Xu et al., 2015), parsing (Vinyals et al., 2015a), dialogue response generation (Li et al., 2016a; Serban et al., 2016) and sentence summarization (Rush et al., 2015; Chopra et al., 2016). In particular, in this pa- per we focus on sentence summarization, which as ∗Now at Preferred Networks. † This work was done when the author was at the Nara In- stitute of Science and Technology. 1Available at https://github.com/kiyukuta/lencon. its name suggests, consists of generating shorter ver- sions of sentences for applications such as document summarization (Nenkova and McKeown, 2011) or headline generation (Dorr et al., 2003). Recently, Rush et al. (2015) automatically constructed large training data for sentence summarization, and this has led to the rapid development of neural sentence summarization (NSS) or neural headline generation (NHG) models. There are already many studies that address this task (Nallapati et al., 2016; Ayana et al., 2016; Ranzato et al., 2015; Lopyrev, 2015; Gulcehre et al., 2016; Gu et al., 2016; Chopra et al., 2016). One of the essential properties that text summa- rization systems should have is the ability to gen- erate a summary with the desired length. Desired lengths of summaries strongly depends on the scene of use, such as the granularity of information the user wants to understand, or the monitor size of the device the user has. The length also depends on the amount of information contained in the given source document. Hence, in the traditional setting of text summarization, both the source document and the desired length of the summary will be given as input to a summarization system. However, methods for controlling the output sequence length of encoder- decoder models have not been investigated yet, de- spite their importance in these settings. In this paper, we propose and investigate four methods for controlling …

Content truncated to 500 words