Instance-Based Question Answering: A Data-Driven Approach

Instance-Based Question Answering: A Data-Driven Approach Lucian Vlad Lita Carnegie Mellon University llita@cs.cmu.edu Jaime Carbonell Carnegie Mellon University jgc@cs.cmu.edu Abstract Anticipating the availability of large question- answer datasets, we propose a principled, data- driven Instance-Based approach to Question An- swering. Most question answering systems incor- porate three major steps: classify questions accord- ing to answer types, formulate queries for document retrieval, and extract actual answers. Under our ap- proach, strategies for answering new questions are directly learned from training data. We learn mod- els of answer type, query content, and answer ex- traction from clusters of similar questions. We view the answer type as a distribution, rather than a class in an ontology. In addition to query expansion, we learn general content features from training data and use them to enhance the queries. Finally, we treat answer extraction as a binary classification problem in which text snippets are labeled as correct or in- correct answers. We present a basic implementation of these concepts that achieves a good performance on TREC test data. 1 Introduction Ever since Question Answering (QA) emerged as an active research field, the community has slowly diversified question types, increased question com- plexity, and refined evaluation metrics - as reflected by the TREC QA track (Voorhees, 2003). Starting from successful pipeline architectures (Moldovan et al., 2000; Hovy et al., 2000), QA systems have re- sponded to changes in the nature of the QA task by incorporating knowledge resources (Hermjakob et al., 2000; Hovy et al., 2002), handling additional types of questions, employing complex reasoning mechanisms (Moldovan et al., 2003; Nyberg et al., 2003), tapping into external data sources such as the Web, encyclopedias, databases (Dumais et al., 2002; Xu et al., 2003), and merging multiple agents and strategies into meta-systems (Chu-Carroll et al., 2003; Burger et al., 2002). In recent years, learning components have started to permeate Question Answering (Clarke et al., 2003; Ravichandran et al., 2003; Echihabi and Marcu, 2003). Although the field is still domi- nated by knowledge-intensive approaches, compo- nents such as question classification, answer extrac- tion, and answer verification are beginning to be ad- dressed through statistical methods. At the same time, research efforts in data acquisition promise to deliver increasingly larger question-answer datasets (Girju et al., 2003; Fleischman et al., 2003). More- over, Question Answering is expanding to different languages (Magnini et al., 2003) and domains other than news stories (Zweigenbaum, 2003). These trends suggest the need for principled, statisti- cally based, easily re-trainable, language indepen- dent QA systems that take full advantage of large amounts of training data. We propose an instance-based, data-driven ap- proach to Question Answering. Instead of classify- ing questions according to limited, predefined on- tologies, we allow training data to shape the strate- gies for answering new questions. Answer mod- els, query content models, and extraction models are also learned directly from training data. We present a basic implementation of these concepts and eval- uate the performance. 2 Motivation Most existing Question Answering systems classify new questions …

Content truncated to 500 words