Learning to Identify Single-Snippet Answers to Definition Questions

Learning to Identify Single-Snippet Answers to Definition Questions Spyridoula MILIARAKI and Ion ANDROUTSOPOULOS Department of Informatics, Athens University of Economics and Business Patission 76, GR-104 34, Athens, Greece Abstract We present a learning-based method to identify single-snippet answers to definition questions in question answering systems for document collections. Our method combines and extends two previous techniques that were based mostly on manually crafted lexical patterns and WordNet hypernyms. We train a Support Vector Machine (SVM) on vectors comprising the verdicts or attributes of the previous techniques, and additional phrasal attributes that we acquire automatically. The SVM is then used to identify and rank single 250-character snippets that contain answers to definition questions. Experimental results indicate that our method clearly outperforms the techniques it builds upon. 1 Introduction Since the introduction of the TREC QA track (Voorhees, 2001), question answering systems for document collections have attracted a lot of attention. The goal is to return from the collection text snippets (eg., 50 or 250 characters long) or exact answers (e.g., names, dates) that answer natural language questions submitted by users. A typical system first classifies the question into one of several categories (questions asking for locations, persons, etc.), producing expectations of types of named entities that must be present in the answer (location names, person names, etc.). Using the question terms as a query, an information retrieval (IR) system identifies possibly relevant passages in the collection, often after query expansion (e.g., adding synonyms). Snippets of these passages are then selected and ranked, based on criteria such as whether or not they contain the expected types of named entities, the percentage of question words in each snippet, the percentage of words that also occur in other candidate snippets, etc. The system reports the most highly-ranked snippets, or, in the case of exact answers, named entities of the required type therein. Unfortunately, the approach highlighted above falls short with questions that do not generate expectations of particular types of named entities and contain very few non-stop-words. Definition questions (e.g. “What is a nanometer?”, “Who was Duke Ellington?”) have both properties, and are particularly common. In TREC-2001, where the distribution of question types reflected that of real user logs, 27% of the questions were requests for definitions. Hence, techniques to handle this category of questions are very important. We propose a new method to answer definition questions, that combines and extends the technique of Prager et al. (2001, 2002), which relied on WordNet hypernyms, and that of Joho et al. (2001, 2002), which relied on manually crafted lexical patterns, sentence position, and word co- occurrence across candidate answers. We train an SVM (Schölkopf and Smola, 2002) on vectors whose attributes include the verdict of Prager et al.’s method, the attributes of Joho et al., and additional phrasal attributes that we acquire automatically. The SVM is then used to identify and rank 250-character snippets, each intended to contain a stand-alone definition of a given term, much as in TREC QA tasks prior to 2003. In …

Content truncated to 500 words