The Use of Dynamic Segment Scoring for Language-Independent Question Answering

The Use of Dynamic Segment Scoring for Language-Independent Question Answering � Daniel Pack � and Clifford Weinstein MIT Lincoln Laboratory 244 Wood Street Lexington, Massachusettes dpack@ll.mit.edu cjw@ll.mit.edu ABSTRACT This paper presents a novel language-independent question/answering (Q/A) system based on natural language processing techniques, shallow query understanding, dynamic sliding window techniques, and statistical proximity distribution matching techniques. The per- formance of the proposed system using the latest Text REtrieval Conference (TREC-8) data was comparable to results reported by the top TREC-8 contenders. Keywords Question/Answer, Natural Language Processing, Query Understand- ing, Dynamic Sliding Window, Proximity Distribution 1. INTRODUCTION Over the past decade, the TREC community has invested its ef- forts on and advanced technologies of automatic information re- trieval systems. Recently, the same community decided to divide the traditional information retrieval task to several so called tracks: the cross-language information retrieval track, the filtering track, the interactive track, the question and answering track, the query track, the spoken document retrieval track, and the web track[6]. The decision is mainly due to the mature technologies in the tradi- tional information retrieval field and the desire to expand the tech- nologies to additional areas of interest. The goal of the question and answering track is the development of systems that generate concise answers to user queries. This goal is similar in nature to the goal of a traditional information retrieval system where relevant � This work was funded by DARPA under Air Force Contract F19628-00-c-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and do not necessarily represent the views of the agency or the US Air Force. � Daniel Pack is an associate professor of Electrical Engineering from the Air Force Academy on his sabbatical leave. . documents are extracted for user queries; users are then required to read through the selected documents to find answers. In a ques- tion answering system, it is the system's responsibility to find the answers to queries. Queries Data Q/A System A B Concept Tagging Stemming POS Tagging Preprocessing Query Processing C D Dynamic Sliding Windows for Sizing Segments Keywords/POS/Q-Concept Weighted Matching Matching Phrases Based on Statistical Distributions Final Selection Frame Final Answers Final Answer Formulation Extraction of Candidate Segments C E Proximity Computation Query Identification Concept Tagging POS Tagging Stemming Preprocessing Translation Module Answers F Translation Module Data Processing Figure 1: The Question and Answering System Architecture In this paper, we present a Q/A system that combines (1) natural language processing techniques, (2) query understanding, (3) dy- namic sliding window techniques, and (4) keyword distance prox- imity distribution matching techniques for a language-independent question/answering system. The system architecture is shown in Figure 1. We call the system language-independent since the sys- tem architecture remains the same regardless of any particular lan- guage used. The only requirement is to have a translation module at the front end and the back end of our system. Developing such systems is becoming increasingly important as the diverse commu- nities across national boundaries are brought together through the internet. …

Content truncated to 500 words