Reducing Odd Generation from Neural Headline Generation

Reducing Odd Generation from Neural Headline Generation Shun Kiyono 1∗ Sho Takase 2 Jun Suzuki 1,2,4 Naoaki Okazaki 3 Kentaro Inui 1,4 Masaaki Nagata 2 1 Tohoku University 3 Tokyo Institute of Technology2 NTT Communication Science Laboratories 4 RIKEN Center for Advanced Intelligence Project {kiyono,jun.suzuki,inui}@ecei.tohoku.ac.jp, {takase.sho, nagata.masaaki}@lab.ntt.co.jp, okazaki@c.titech.ac.jp Abstract The Encoder-Decoder model is widely used in natural language generation tasks. However, the model sometimes suffers from repeated re- dundant generation, misses important phrases, and includes irrelevant entities. Toward solv- ing these problems we propose a novel source- side token prediction module. Our method jointly estimates the probability distributions over source and target vocabularies to capture the correspondence between source and target tokens. Experiments show that the proposed model outperforms the current state-of-the-art method in the headline generation task. We also show that our method can learn a reason- able token-wise correspondence without know- ing any true alignment1. 1 Introduction The Encoder-Decoder model with the attention mech- anism (EncDec) (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) has been an epoch-making development that has led to great progress being made in many natural language generation tasks, such as machine translation (Bah- danau et al., 2015), dialog generation (Shang et al., 2015), and headline generation (Rush et al., 2015). Today, EncDec and its variants are widely used as the predominant baseline method in these tasks. ∗This work is a product of collaborative research program of Tohoku University and NTT Communication Science Labora- tories. 1Our code for reproducing the experiments is available at https://github.com/butsugiri/UAM Unfortunately, as often discussed in the commu- nity, EncDec sometimes generates sentences with repeating phrases or completely irrelevant phrases and the reason for their generation cannot be inter- preted intuitively. Moreover, EncDec also sometimes generates sentences that lack important phrases. We refer to these observations as the odd generation prob- lem (odd-gen) in EncDec. The following table shows typical examples of odd-gen actually generated by a typical EncDec. (1) Repeating Phrases Gold: duran duran group fashionable again EncDec: duran duran duran duran (2) Lack of Important Phrases Gold: graf says goodbye to tennis due to injuries EncDec: graf retires (3) Irrelevant Phrases Gold: u.s. troops take first position in serb-held bosnia EncDec: precede sarajevo This paper tackles for reducing the odd-gen in the task of abstractive summarization. In machine trans- lation literature, coverage (Tu et al., 2016; Mi et al., 2016) and reconstruction (Tu et al., 2017) are promis- ing extensions of EncDec to address the odd-gen. These models take advantage of the fact that machine translation is the loss-less generation (lossless-gen) task, where the semantic information of source- and target-side sequence is equivalent. However, as dis- cussed in previous studies, abstractive summarization is a lossy-compression generation (lossy-gen) task. Here, the task is to delete certain semantic informa- tion from the source to generate target-side sequence. PACLIC 32 289 32nd Pacific Asia Conference on Language, Information and Computation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors Therefore the models …

Content truncated to 500 words