Ranking suspected answers to natural language questions using predictive annotation

Ranking suspected answers to natural language questions using predictive annotation Dragomir R. Radev" School of Information University of Michigan Ann Arbor, MI 48103 radev@umich, edu John Prager TJ Watson Research Center IBM Research Division Hawthorne, NY 10532 jprager@us.ibm.com Valerie Samn* Teachers College Columbia University New York, NY 10027 vsll5@columbia.edu Abstract In this paper, we describe a system to rank sus- pected answers to natural language questions. We process both corpus and query using a new technique, predictive annotation, which aug- ments phrases in texts with labels anticipating their being targets of certain kinds of questions. Given a natural language question, our IR sys- tem returns a set of matching passages, which we then rank using a linear function of seven predictor variables. We provide an evaluation of the techniques based on results from the TREC Q&A evaluation in which our system partici- pated. 1 Introduction Question Answering is a task that calls for a combination of techniques from Information Re- trieval and Natural Language Processing. The former has the advantage of years of develop- ment of efficient techniques for indexing and searching large collections of data, but lacks of any meaningful treatment of the semantics of the query or the texts indexed. NLP tackles the semantics, but tends to be computationally expensive. We have attempted to carve out a middle ground, whereby we use a modified IR system augmented by shallow NL parsing. Our ap- proach was motivated by the following problem with traditional IR systems. Suppose the user asks "Where did <some event> happen?". If the system does no pre-processing of the query, then "where" will be included in the bag of words submitted to the search engine, but this will not be helpful since the target text will be unlikely to contain the word "where". If the word is stripped out as a stop-word, then * The work presented in this paper was performed while the first and third authors were at 1BM Research. the search engine will have no idea that a lo- cation is sought. Our approach, called predic- tive annotation, is to augment the query with semantic category markers (which we call QA- Tokens), in this case with the PLACES to- ken, and also to label with QA-Tokens all oc- currences in text that are recognized entities, (for example, places). Then traditional bag-of- words matching proceeds successfully, and will return matching passages. The answer-selection process then looks for and ranks in these pas- sages occurrences of phrases containing the par- ticular QA-Token(s) from the augmented query. This classification of questions is conceptually similar to the query expansion in (Voorhees, 1994) but is expected to achieve much better performance since potentially matching phrases in text are classified in a similar and synergistic way. Our system participated in the official TREC Q&A evaluation. For 200 questions in the eval- uation set, we were asked to provide a list of 50-byte and 250-byte extracts from a 2-GB cor- pus. The results are shown in Section 7. Some techniques …

Content truncated to 500 words