Content Models for Survey Generation: A Factoid-Based Evaluation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 441–450, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Content Models for Survey Generation: A Factoid-Based Evaluation Rahul Jha⋆, Catherine Finegan-Dollak⋆, Reed Coke⋆, Ben King⋆, Dragomir Radev⋆† ⋆ Department of EECS, University of Michigan, USA † School of Information, University of Michigan, USA {rahuljha,cfdollak,reedcoke,benking,radev}@umich.edu Abstract We present a new factoid-annotated dataset for evaluating content models for scientific survey article generation containing 3,425 sentences from 7 topics in natural language processing. We also introduce a novel HITS-based content model for automated survey article gen- eration called HITSUM that exploits the lexical network structure between sen- tences from citing and cited papers. Using the factoid-annotated data, we conduct a pyramid evaluation and compare HITSUM with two previous state-of-the-art content models: C-Lexrank, a network based con- tent model, and TOPICSUM, a Bayesian content model. Our experiments show that our new content model captures useful survey-worthy information and outper- forms C-Lexrank by 4% and TOPICSUM by 7% in pyramid evaluation. 1 Introduction Survey article generation is the task of automat- ically building informative surveys for scientific topics. Given the rapid growth of publications in scientific fields, the development of such systems is crucial as human-written surveys exist for a lim- ited number of topics and get outdated quickly. In this paper, we investigate content models for extracting survey-worthy information from scien- tific papers. Such models are an essential com- ponent of any system for automatic survey arti- cle generation. Earlier work in the area of survey article generation has investigated content mod- els based on lexical networks (Mohammad et al., 2009; Qazvinian and Radev, 2008). These mod- els take as input citing sentences that describe important papers on the topic and assign them a salience score based on centrality in a lexical net- work formed by the input citing sentences. In this Factoid Weight Question Answering answer extraction 6 question classification 6 definition of question answering 5 TREC QA track 5 information retrieval 5 Dependency Parsing non-projective dependency structures / trees 6 projectivity / projective dependency trees 6 deterministic parsing approaches: Nivre’s algorithm 5 terminology: head - dependent 4 grammar driven approaches for dependency parsing 4 Table 1: Sample factoids from the topics of ques- tion answering and dependency parsing along with their factoid weights. paper, we propose a new content model based on network structure previously unexplored for this task that exploits the lexical relationship between citing sentences and the sentences from the origi- nal papers that they cite. Our new formulation of the lexical network structure fits nicely with the hubs and authorities model for identifying impor- tant nodes in a network (Kleinberg, 1999), leading to a new content model called HITSUM. In addi- tion to this new content model, we also describe how Bayesian content models previously explored in the news domain can be adapted for the content modeling task for survey generation. For the task …

Content truncated to 500 words