Diverse Keyphrase Generation with Neural Unlikelihood Training

Proceedings of the 28th International Conference on Computational Linguistics, pages 5271–5287 Barcelona, Spain (Online), December 8-13, 2020 5271 Diverse Keyphrase Generation with Neural Unlikelihood Training Hareesh Bahuleyan† Layla El Asri† †Borealis AI, Montreal, Canada layla.elasri@borealisai.com hareeshbahuleyan@gmail.com Abstract In this paper, we study sequence-to-sequence (S2S) keyphrase generation models from the perspective of diversity. Recent advances in neural natural language generation have made possible remarkable progress on the task of keyphrase generation, demonstrated through improvements on quality metrics such as F1-score. However, the importance of diversity in keyphrase generation has been largely ignored. We first analyze the extent of information redundancy present in the outputs generated by a baseline model trained using maximum likelihood estimation (MLE). Our findings show that repetition of keyphrases is a major issue with MLE training. To alleviate this issue, we adopt neural unlikelihood (UL) objective for training the S2S model. Our version of UL training operates at (1) the target token level to discourage the generation of repeating tokens; (2) the copy token level to avoid copying repetitive tokens from the source text. Further, to encourage better model planning during the decoding process, we incorporate K-step ahead token prediction objective that computes both MLE and UL losses on future tokens as well. Through extensive experiments on datasets from three different domains we demonstrate that the proposed approach attains considerably large diversity gains, while maintaining competitive output quality.1 1 Introduction Automatic keyphrase generation is the task of generating single or multi-word lexical units that provides readers with high level information about the key ideas or important topics described in a given source text. Apart from an information summarization perspective, this task has applications in various downstream natural language processing tasks such as text classification (Liu et al., 2009), document clustering (Hammouda et al., 2005) and information retrieval (Nguyen and Kan, 2007a). Traditionally, keyphrases (KPs) were extracted from source documents by retrieving and ranking a set of candidate phrases through rule based approaches. With recent advances in neural natural language generation and availability of larger training corpora, this problem is formulated under a sequence-to- sequence (S2S) modelling framework (Sutskever et al., 2014). This approach has an advantage that it can generate new and meaningful keyphrases which may be absent in the source text. The earliest work in this direction was by Meng et al. (2017), who train a S2S model to generate one keyphrase at a time. At inference time, they decode with beam sizes as high as 200, to generate a large number of KPs and finally de-duplicate the outputs. However, this is computationally expensive and wasteful because only < 5% of such KPs were found to be unique (Yuan et al., 2020). An alternative approach is to train a S2S model to generate multiple keyphrases in a sequential manner, where the output KPs are separated by a pre-defined delimiter token. This method has an added benefit that the model automatically learns to generate a variable number of keyphrases depending on the input, instead of a user-specified fixed …

Content truncated to 500 words