The Effectiveness of Dictionary and Web-Based Answer Reranking

. The Effectiveness of Dictionary and Web-Based Answer Reranking Chin-Yew Lin Information Sciences Institute University of Southern California 4676 Admiralty Way Marina del Rey, CA 90292 cyl@isi.edu Abstract We describe an in-depth study of using a dictionary (WordNet) and web search engines (Altavista, MSN, and Google) to boost the performance of an automated question answering system, Webclopedia, in answering definition questions. The results indicate applying dictionary and web-based answer reranking together increase the performance of Webclopedia on a set of 102 TREC-10 definition questions by 25% in mean reciprocal rank score and 14% in finding answers in the top 5. 1 Introduction In an attempt to further progress in information retrieval research, the Text REtrieval Conference (TREC) sponsored by the National Institute of Standards and Technology (NIST) started a series of large-scale evaluations of domain independent automated question answering systems in TREC-8 (Voorhees 2000) and continued in TREC-9 and TREC-10. NTCIR (NII-NACSIS Test Collection for IR Systems, TREC’s counterpart in Japan) initiated its question answering evaluation effort, Question Answering Challenge (QAC) in 2001 (Fukumoto et al. 2001). Research systems participating in TRECs and the coming QAC focused on the problem of answering closed-class questions that have short fact-based answers (“factoids”) from a large collection of text. These systems bear a similar structure: (1) Question analysis – identify question keywords to be submitted to search engines (local or web), recognize question types, and suggest expected answer types. Although most systems rely on a taxonomy of expected answer types, the number of nodes in the taxonomy varies widely from single digits to a few thousands. For example, Abney et al. (2000) used 5; Ittycheriah et al. (2001), 31; Hovy et al. (2001), 140; Harabagiu et al. (2001), 8,797. These taxonomies were mostly based on named entities and WordNet (Fellbaum 1998). Special types such definition questions (ex: “What is an atom?”) were added as necessary. (2) Passage or Sentence retrieval – this aims to provide a text pool of manageable size for extracting candidate answers. Most top performing systems in TRECs use their own retrieval methods for passages (Brill et al. 2001; Clarke et al. 2001; Harabagiu et al. 2001) or sentences (Hovy et al. 2001). (3) Candidate answer extraction – extract candidate answers according to answer types. If the expected answer types are typical named entities, information extraction engines (Bikel et al. 1999, Srihari and Li 2000) are used to extract candidate answers. Otherwise special answer patterns are used to pinpoint answers. For example, Soubbotin and Soubbotin (2001) create a set of 6 answer patterns for definition questions. (4) Answer ranking – assign scores to candidate answers according to their frequency in top ranked passages (Abney et al. 2000; Clarke et al. 2001), similarity to candidate answers extracted from external sources such as the web (Brill et al. 2001; Buchholz 2001) or WordNet (Harabagiu et al. 2001; Hovy et al. 2001), density, distance, or order of question keywords around the candidates, similarity between the dependency structures of questions and candidate answers (Harabagiu …

Content truncated to 500 words