Positional Encoding to Control Output Sequence Length

Proceedings of NAACL-HLT 2019, pages 3999–4004 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3999 Positional Encoding to Control Output Sequence Length Sho Takase Naoaki Okazaki Tokyo Institute of Technology {sho.takase@nlp.c, okazaki@c}.titech.ac.jp Abstract Neural encoder-decoder models have been suc- cessful in natural language generation tasks. However, real applications of abstractive sum- marization must consider additional constraint that a generated summary should not exceed a desired length. In this paper, we propose a simple but effective extension of a sinusoidal positional encoding (Vaswani et al., 2017) to enable neural encoder-decoder model to pre- serves the length constraint. Unlike in previ- ous studies where that learn embeddings repre- senting each length, the proposed method can generate a text of any length even if the tar- get length is not present in training data. The experimental results show that the proposed method can not only control the generation length but also improve the ROUGE scores. 1 Introduction Neural encoder-decoder models have been success- fully applied to various natural language genera- tion tasks including machine translation (Sutskever et al., 2014), summarization (Rush et al., 2015), and caption generation (Vinyals et al., 2015). Still, it is necessary to control the output length for ab- stractive summarization, which generates a sum- mary for a given text while satisfying a space con- straint. In fact, Figure 1 shows a large variance in output sequences produced by a widely used encoder-decoder model (Luong et al., 2015), which has no mechanism for controlling the length of the output sequences. Fan et al. (2018) trained embeddings that corre- spond to each output length to control the output sequence length. Since the embeddings for differ- ent lengths are independent, it is hard to generate a sequence of the length that is infrequent in training data. Thus, a method that can model any lengths continuously is required. Figure 1: Difference in number of characters between correct headlines and outputs of a widely used LSTM encoder-decoder (Luong et al., 2015) which is trained on sentence-headline pairs created by Rush et al. (2015) from the annotated English Gigaword corpus. The dif- ference was investigated for 3,000 sentence-headline pairs randomly sampled from the test splits. Kikuchi et al. (2016) proposed two learning based methods for an LSTM encoder-decoder: LenEmb and LenInit. LenEmb inputs an embed- ding representing the remaining length in each de- coding step. Since this approach also prepares em- beddings for each length independently, it suffers from the same problem as that in Fan et al. (2018). On the other hand, LenInit can handle arbitrary lengths because it combines the scalar value of a desired length with a trainable embedding. LenInit initializes the LSTM cell of the decoder with the embedding depending on the scalar value of the desired length. Liu et al. (2018) incorporated such scalar values into the initial state of the decoder in a CNN encoder-decoder. These approaches deal with any length but it is reasonable to incorporate the distance to the desired terminal position …

Content truncated to 500 words