AttentionRank: Unsupervised Keyphrase Extraction using Self and Cross Attentions

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1919–1928 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1919 AttentionRank: Unsupervised keyphrase Extraction using Self and Cross Attentions Haoran Ding and Xiao Luo Purdue School of Enigneering and Technology, IUPUI, USA hd10@iu.edu, luo25@iupui.edu Abstract Keyword or keyphrase extraction is to iden- tify words or phrases presenting the main top- ics of a document. This paper proposes the AttentionRank, a hybrid attention model, to identify keyphrases from a document in an un- supervised manner. AttentionRank calculates self-attention and cross-attention using a pre- trained language model. The self-attention is designed to determine the importance of a candidate within the context of a sentence. The cross-attention is calculated to identify the semantic relevance between a candidate and sentences within a document. We evalu- ate the AttentionRank on three publicly avail- able datasets against seven baselines. The re- sults show that the AttentionRank is an effec- tive and robust unsupervised keyphrase extrac- tion model on both long and short documents. Source code is available on Github 1. 1 Introduction A vast amount of scientific or non-scientific arti- cles are published online every year. Although some of them have associated keyphrases speci- fied, which made them easy to index and search, a considerable amount of them has no keyphrase defined, making indexing and information retrieval challenging. Given so many articles available, it is not feasible to manually extract keyphrases from each. Automate the keyphrases extraction becomes crucial. Keyphrase extraction has immense value to the downstream text mining tasks, such as text seg- mentation, text summarization, query expansion, indexing, and so on. Keyphrase extraction methods can be supervised or unsupervised. Traditional supervision methods use decision tree (Turney, 2000) or naive Bayes (Witten et al., 2005) to identify whether the in- put word/phrase is a keyphrase. With the advanc- ing of neural networks, various supervised deep 1https://github.com/hd10-iupui/AttentionRank learning models (Meng et al., 2017; Alzaidy et al., 2019; Sun et al., 2019) are developed to extract keyphrases. The supervised method requires a large labeled training dataset and is often domain- specific, whereas unsupervised methods do not need labeled datasets. The traditional unsupervised methods use statistical and graph-based approaches. The statistical-based methods (Beliga et al., 2016; Rose et al., 2010; Campos et al., 2018) utilize the candidate position, frequency, length, and capi- talization to determine the importance of a word. The graph-based approaches (Wan and Xiao, 2008; Gollapalli and Caragea, 2014) construct a graph with the candidates as nodes. The edges indicate the similarity or co-occurrences of the candidates. Graph-based algorithms can be applied to calcu- late the importance of the nodes (candidates) on the graph. Driven by recent deep language mod- els for natural language processing and text analy- sis, text embedding-based or mixed statistical and embedding-based unsupervised keyphrase extrac- tion methods have emerged, such as EmbedRank (Bennani-Smires et al., 2018), SIFRank (Sun et al., 2020), and KeyGames (Saxena et al., 2020). In this research, we propose an attention-based unsupervised model – AttentionRank for …

Content truncated to 500 words