Automatic Set Expansion for List Question Answering

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 947–954, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Automatic Set Expansion for List Question Answering Richard C. Wang, Nico Schlaefer, William W. Cohen, and Eric Nyberg Language Technologies Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh PA 15213 {rcwang,nico,wcohen,ehn}@cs.cmu.edu Abstract This paper explores the use of set expan- sion (SE) to improve question answering (QA) when the expected answer is a list of entities belonging to a certain class. Given a small set of seeds, SE algorithms mine textual re- sources to produce an extended list including additional members of the class represented by the seeds. We explore the hypothesis that a noise-resistant SE algorithm can be used to extend candidate answers produced by a QA system and generate a new list of answers that is better than the original list produced by the QA system. We further introduce a hybrid ap- proach which combines the original answers from the QA system with the output from the SE algorithm. Experimental results for several state-of-the-art QA systems show that the hy- brid system performs better than the QA sys- tems alone when tested on list question data from past TREC evaluations. 1 Introduction Question answering (QA) systems are designed to retrieve precise answers to questions posed in nat- ural language. A list question expects a list as its answer, e.g. Name the coffee-producing countries in South America. The ability to answer list questions has been tested as part of the yearly TREC QA eval- uation (Dang et al., 2006; Dang et al., 2007). This paper focuses on the use of set expansion to improve list question answering. A set expansion (SE) algo- rithm receives as input a few members of a class or set, and mines various textual resources (e.g. web pages) to produce an extended list including addi- tional members of the class or set that are not in the input. A well-known online SE system is Google Sets1. This system is publicly accessible, but since it is a proprietary system that might be changed at any time, its results cannot be replicated reliably. We ex- plore the hypothesis that a SE algorithm, when care- fully designed to handle noisy inputs, can be applied to the output from a QA system to produce an overall list of answers for a given question that is better than the answers produced by the QA system itself. We propose to exploit large, redundant sources of struc- tured and/or semi-structured data and use linguistic analysis to seed a shallow analysis of these sources. This is a hard problem since the linguistic evidence used as seeds is noisy. More precisely, we combine the QA system Ephyra (Schlaefer et al., 2007) with the SE system SEAL (Wang and Cohen, 2007) to create a hybrid approach that performs better than either system by itself when tested on data from the TREC 13-15 evaluations. In addition, we apply our SE algorithm to answers generated …

Content truncated to 500 words