Enhanced Answer Type Inference from Questions using Sequential Models

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 315–322, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Enhanced Answer Type Inference from Questions using Sequential Models Vijay Krishnan and Sujatha Das and Soumen Chakrabarti∗ Computer Science and Engineering Department, IIT Bombay, India Abstract Question classification is an important step in factual question answering (QA) and other dialog systems. Several at- tempts have been made to apply statistical machine learning approaches, including Support Vector Machines (SVMs) with sophisticated features and kernels. Curi- ously, the payoff beyond a simple bag-of- words representation has been small. We show that most questions reveal their class through a short contiguous token subse- quence, which we call its informer span. Perfect knowledge of informer spans can enhance accuracy from 79.4% to 88% using linear SVMs on standard bench- marks. In contrast, standard heuristics based on shallow pattern-matching give only a 3% improvement, showing that the notion of an informer is non-trivial. Us- ing a novel multi-resolution encoding of the question’s parse tree, we induce a Con- ditional Random Field (CRF) to identify informer spans with about 85% accuracy. Then we build a meta-classifier using a linear SVM on the CRF output, enhancing accuracy to 86.2%, which is better than all published numbers. 1 Introduction An important step in factual question answering (QA) and other dialog systems is to classify the question (e.g., Who painted Olympia?) to the antic- ipated type of the answer (e.g., person). This step is called “question classification” or “answer type identification”. The answer type is picked from a hand-built tax- onomy having dozens to hundreds of answer types (Harabagiu et al., 2000; Hovy et al., 2001; Kwok et al., 2001; Zheng, 2002; Dumais et al., 2002). QA ∗ soumen@cse.iitb.ac.in systems can use the answer type to short-list answer tokens from passages retrieved by an information re- trieval (IR) subsystem, or use the type together with other question words to inject IR queries. Early successful QA systems used manually- constructed sets of rules to map a question to a type, exploiting clues such as the wh-word (who, where, when, how many) and the head of noun phrases associated with the main verb (what is the tallest mountain in . . .). With the increasing popularity of statistical NLP, Li and Roth (2002), Hacioglu and Ward (2003) and Zhang and Lee (2003) used supervised learning for question classification on a data set from UIUC that is now standard1. It has 6 coarse and 50 fine answer types in a two-level taxonomy, together with 5500 training and 500 test questions. Webclopedia (Hovy et al., 2001) has also published its taxonomy with over 140 types. The promise of a machine learning approach is that the QA system builder can now focus on de- signing features and providing labeled data, rather than coding and maintaining complex heuristic rule- bases. The data sets and learning systems quoted above have made question classification a well- defined and non-trivial subtask of …

Content truncated to 500 words