Supervised Keyphrase Extraction as Positive Unlabeled Learning

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1924–1929, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Supervised Keyphrase Extraction as Positive Unlabeled Learning Lucas Sterckx∗, Cornelia Caragea†, Thomas Demeester∗, Chris Develder∗ ∗Ghent University – iMinds, Ghent, Belgium {lusterck,tdmeeste,cdvelder}@intec.ugent.be †University of North Texas, Texas, USA cornelia.caragea@unt.edu Abstract The problem of noisy and unbalanced train- ing data for supervised keyphrase extraction results from the subjectivity of keyphrase as- signment, which we quantify by crowdsourc- ing keyphrases for news and fashion magazine articles with many annotators per document. We show that annotators exhibit substantial disagreement, meaning that single annotator data could lead to very different training sets for supervised keyphrase extractors. Thus, an- notations from single authors or readers lead to noisy training data and poor extraction per- formance of the resulting supervised extractor. We provide a simple but effective solution to still work with such data by reweighting the importance of unlabeled candidate phrases in a two stage Positive Unlabeled Learning set- ting. We show that performance of trained keyphrase extractors approximates a classi- fier trained on articles labeled by multiple an- notators, leading to higher average F1scores and better rankings of keyphrases. We ap- ply this strategy to a variety of test collec- tions from different backgrounds and show improvements over strong baseline models. 1 Introduction Keyphrase extraction is the task of extracting a se- lection of phrases from a text document to concisely summarize its contents. Applications of keyphrases range from summarization (D’Avanzo et al., 2004) to contextual advertisement (Yih et al., 2006) or sim- ply as aid for navigation through large text corpora. Existing work on automatic keyphrase extraction can be divided in supervised and unsupervised ap- proaches. While unsupervised approaches are do- main independent and do not require labeled train- ing data, supervised keyphrase extraction allows for more expressive feature design and is reported to outperform unsupervised methods on many occa- sions (Kim et al., 2012; Caragea et al., 2014). A requirement for supervised keyphrase extractors is the availability of labeled training data. In literature, training collections for supervised keyphrase extrac- tion are generated in different settings. In these col- lections, keyphrases for text documents are either supplied by the authors or their readers. In the first case, authors of academic papers or news articles as- sign keyphrases to their content to enable fast index- ing or to allow for the discovery of their work in electronic libraries (Frank et al., 1999; Hulth, 2003; Bulgarov and Caragea, 2015). Other collections are created by crowdsourcing (Marujo et al., 2012) or based on explicit deliberation by a small group of readers (Wan and Xiao, 2008). A minority of test collections provide multiple opinions per document, but even then the amount of opinions per document is kept minimal (Nguyen and Kan, 2007). The traditional procedure for supervised keyphrase extraction is reformulating the task as a binary classification of keyphrase candidates. Supervision for keyphrase extraction faces several shortcomings. Candidate phrases (generated in a separate …

Content truncated to 500 words