Reranking Answers for Definitional QA Using Language Modeling

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 1081–1088, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Reranking Answers for Definitional QA Using Language Modeling Yi Chen School of Software Engi- neering Chongqing University Chongqing, China, 400044 126cy@126.com Ming Zhou Microsoft Research Asia 5F Sigma Center, No.49 Zhichun Road, Haidian Bejing, China, 100080 mingzhou@microsoft.com Shilong Wang College of Mechanical En- gineering Chongqing University Chongqing, China, 400044 slwang@cqu.edu.cn Abstract* Statistical ranking methods based on cen- troid vector (profile) extracted from ex- ternal knowledge have become widely adopted in the top definitional QA sys- tems in TREC 2003 and 2004. In these approaches, terms in the centroid vector are treated as a bag of words based on the independent assumption. To relax this as- sumption, this paper proposes a novel language model-based answer reranking method to improve the existing bag-of- words model approach by considering the dependence of the words in the centroid vector. Experiments have been conducted to evaluate the different dependence models. The results on the TREC 2003 test set show that the reranking approach with biterm language model, significantly outperforms the one with the bag-of- words model and unigram language model by 14.9% and 12.5% respectively in F-Measure(5). 1 Introduction In recent years, QA systems in TREC (Text RE- trieval Conference) have made remarkable pro- gress (Voorhees, 2002). The task of TREC QA before 2003 has mainly focused on the factoid questions, in which the answer to the question is a number, a person name, or an organization name, or the like. Questions like “Who is Colin Powell?” or “What is mold?” are definitional questions *This work was finished while the first author was visiting Microsoft Research Asia during March 2005-March 2006 as a component of the project of AskBill Chatbot led by Dr. Ming Zhou. (Voorhees, 2003). Statistics from 2,516 Fre- quently Asked Questions (FAQ) extracted from Internet FAQ Archives1 show that around 23.6% are definitional questions. This indicates that definitional questions occur frequently and are important question types. TREC started the evaluation for definitional QA in 2003. The defi- nitional QA systems in TREC are required to extract definitional nuggets/sentences that con- tain the highly descriptive information about the question target from a given large corpus. For definitional question, statistical ranking methods based on centroid vector (profile) ex- tracted from external resources, such as the online encyclopedia, are widely adopted in the top systems in TREC 2003 and 2004 (Xu et al., 2003; Blair-Goldensohn et al., 2003; Wu et al., 2004). In these systems, for a given question, a vector is formed consisting of the most frequent co-occurring terms with the question target as the question profile. Candidate answers extracted from a given large corpus are ranked based on their similarity to the question profile. The simi- larity is normally the TFIDF score in which both the candidate answer and the question profile are treated as a bag of words in the framework of Vector Space Model (VSM). …

Content truncated to 500 words