One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961–7975 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7961 One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases Xingdi Yuan†∗ Tong Wang†∗ Rui Meng‡∗ Khushboo Thaker‡ Peter Brusilovsky‡ Daqing He‡ Adam Trischler† †Microsoft Research, Montr´eal ‡School of Computing and Information, University of Pittsburgh {eric.yuan, tong.wang}@microsoft.com rui.meng@pitt.edu Abstract Different texts shall by nature correspond to different number of keyphrases. This desider- atum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modeling and evaluation perspectives. We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. Generation di- versity is further enhanced with two novel tech- niques by manipulating decoder hidden states. In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs. We further propose two evaluation metrics tai- lored towards the variable-number generation. We also introduce a new dataset (STACKEX) that expands beyond the only existing genre (i.e., academic writing) in keyphrase genera- tion tasks. With both previous and new eval- uation metrics, our model outperforms strong baselines on all datasets. 1 Introduction Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that sum- marize the high-level meaning and highlight cer- tain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only dis- tilling high-level information from a document, but also locating specific, important snippets therein. To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). More- over, a given source text is usually associated with ∗ These authors contributed equally. The order is deter- mined by a fidget spinner. Dataset #Train #Valid #Test Mean Var %Pre KP20K ≈514k ≈20k ≈20k 5.3 14.2 63.3% IN S P E C – 1500 500 9.6 22.4 78.5% KR A P I V I N – 1844 460 5.2 6.6 56.2% NUS – - 211 11.5 64.6 51.3% SE MEV A L – 144 100 15.7 15.1 44.5% ST A C KEX ≈298k ≈16k ≈16k 2.7 1.4 57.5% Table 1: Statistics of various datasets. Mean and Var in- dicate the mean and variance of target phrase numbers, %Pre denotes percentage of present keyphrases. a set of multiple keyphrases. Thus, keyphrase gen- eration is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source. Similar to summarization, keyphrase genera- tion is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies (Meng et al., 2017; Chen et al., 2018a; Ye and Wang, 2018; Chen et al., 2018b). Conditioned on a source text, Seq2Seq models generate phrases individually or as …

Content truncated to 500 words