A Practically Unsupervised Learning Method to Identify Single-Snippet Answers to Definition Questions on the Web

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 323–330, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics A Practically Unsupervised Learning Method to Identify Single-Snippet Answers to Definition Questions on the Web Ion Androutsopoulos and Dimitrios Galanis Department of Informatics Athens University of Economics and Business Patission 76, GR-104 34, Athens, Greece Abstract We present a practically unsupervised learning method to produce single-snippet answers to definition questions in ques- tion answering systems that supplement Web search engines. The method exploits on-line encyclopedias and dictionaries to generate automatically an arbitrarily large number of positive and negative definition examples, which are then used to train an SVM to separate the two classes. We show experimentally that the proposed method is viable, that it outperforms the alterna- tive of training the system on questions and news articles from TREC, and that it helps the search engine handle definition questions significantly better. 1 Introduction Question answering (QA) systems for document col- lections typically aim to identify in the collections text snippets (e.g., 50 or 250 characters long) or ex- act answers (e.g., names, dates) that answer natu- ral language questions submitted by their users. Al- though they are commonly evaluated on newspaper archives, as in the TREC QA track, QA systems can also supplement Web search engines, to help them return snippets, as opposed to Web pages, that pro- vide more directly the information users require. Most current QA systems first classify the input question into one of several categories (e.g., ques- tions asking for locations, persons, etc.), producing expectations for types of named entities that must be present in the answer (locations, person names, etc.). Using the question’s terms as a query, an infor- mation retrieval (IR) system identifies relevant doc- uments. Snippets of these documents are then se- lected and ranked, using criteria such as whether or not they contain the expected types of named enti- ties, the percentage of the question’s terms they con- tain, etc. The system then outputs the most highly- ranked snippets, or named entities therein. The approach highlighted above performs poorly with definition questions (e.g., “What is gasohol?”, “Who was Duke Ellington?”), because definition questions do not generate expectations for particular types of named entities, and they typically contain only a single term. Definition questions are particu- larly common; in the QA track of TREC-2001, where the distribution of question types reflected that of real user logs, 27% of the questions were requests for definitions. Of course, answers to many defini- tion questions can be found in on-line encyclopedias and dictionaries.1 There are always, however, new or less widely used terms that are not included in such resources, and this is also true for many names of persons and products. Hence, techniques to dis- cover definitions in ordinary Web pages and other document collections are valuable. Definitions of this kind are often ‘hidden’ in oblique contexts (e.g., “He said that gasohol, a mixture of gasoline and ethanol, has …

Content truncated to 500 words