Language-independent Probabilistic Answer Ranking for Question Answering

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 784–791, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Language-independent Probabilistic Answer Ranking for Question Answering Jeongwoo Ko, Teruko Mitamura, Eric Nyberg Language Technologies Institute School of Computer Science Carnegie Mellon University {jko, teruko, ehn}@cs.cmu.edu Abstract This paper presents a language-independent probabilistic answer ranking framework for question answering. The framework esti- mates the probability of an individual an- swer candidate given the degree of answer relevance and the amount of supporting evi- dence provided in the set of answer candi- dates for the question. Our approach was evaluated by comparing the candidate an- swer sets generated by Chinese and Japanese answer extractors with the re-ranked answer sets produced by the answer ranking frame- work. Empirical results from testing on NT- CIR factoid questions show a 40% perfor- mance improvement in Chinese answer se- lection and a 45% improvement in Japanese answer selection. 1 Introduction Question answering (QA) systems aim at find- ing precise answers to natural language questions from large document collections. Typical QA sys- tems (Prager et al., 2000; Clarke et al., 2001; Harabagiu et al., 2000) adopt a pipeline architec- ture that incorporates four major steps: (1) question analysis, (2) document retrieval, (3) answer extrac- tion and (4) answer selection. Question analysis is a process which analyzes a question and produces a list of keywords. Document retrieval is a step that searches for relevant documents or passages. An- swer extraction extracts a list of answer candidates from the retrieved documents. Answer selection is a process which pinpoints correct answer(s) from the extracted candidate answers. Since the first three steps in the QA pipeline may produce erroneous outputs, the final answer selec- tion step often entails identifying correct answer(s) amongst many incorrect ones. For example, given the question “Which Chinese city has the largest number of foreign financial companies?”, the an- swer extraction component produces a ranked list of five answer candidates: Beijing (AP880603-0268)1, Hong Kong (WSJ920110-0013), Shanghai (FBIS3- 58), Taiwan (FT942-2016) and Shanghai (FBIS3- 45320). Due to imprecision in answer extraction, an incorrect answer (“Beijing”) can be ranked in the first position, and the correct answer (“Shang- hai”) was extracted from two different documents and ranked in the third and the fifth positions. In or- der to rank “Shanghai” in the top position, we have to address two interesting challenges: • Answer Similarity. How do we exploit simi- larity among answer candidates? For example, when the candidates list contains redundant an- swers (e.g., “Shanghai” as above) or several an- swers which represent a single instance (e.g. “U.S.A.” and “the United States”), how much should we boost the rank of the redundant an- swers? • Answer Relevance. How do we identify relevant answer(s) amongst irrelevant ones? This task may involve searching for evi- dence of a relationship between the answer 1Answer candidates are shown with the identifier of the TREC document where they were found. 784 and the answer type or a question key- …

Content truncated to 500 words