Towards semantic-based overlap measures for question-answering

Towards Semantic-Based Overlap Measures for Question Answering Diego Moll´a Centre for Language Technology Macquarie University Sydney, NSW 2109 Tel. +61 2 9850 9531 Fax +61 2 9850 9551 diego@ics.mq.edu.au Abstract In this paper we present an evalua- tion of overlap-based measures of sim- ilarity for sentences in the same lan- guage. The measures include syntactic and semantic information, and to that end they incorporate grammatical re- lations and flat logical forms. A full parser is required to build the above in- formation. Separate extrinsic evalua- tions within the context of question an- swering have been made with two dif- ferent parsers to test the impact of the parser and the overlap measures. 1 Introduction Text-based Question Answering (QA) is a hot research topic and the increasing availability of electronic text will ensure that research in this area will continue for long. Much of the cur- rent research on QA focuses on the development of methodologies for processing relatively large volumes of text. For example, the competition- based QA track of the Text REtrieval Conference (TREC) (Voorhees, 2001) uses more than 3Gb of source text. Competing systems often exploit the data redundancy existing in the source text. Some of them even use the Web to increase the data re- dundancy (Brill et al., 2001; Clarke et al., 2001, for example). These systems typically trade ac- curacy for speed and avoid the use of intensive natural language processing techniques. Most of the current QA systems are based on an architecture like that of Figure 1 (Hirschman and Gaizauskas, 2001; Voorhees, 2001). In an off-line or indexing stage, an indexing module analyses the text documents and creates a set of document images that will be used by the subsequent QA modules. In an on-line stage, a question analy- sis module classifies the question and determines the type of the expected answer. The question analysis module would typically return a list of named-entity types that are compatible with the question (for example a who question typically indicates people or organisations). The module may also produce an image of the question. This image may be similar in format to the document images and can range from a simple bag of words (Cooper and R¨uger, 2000, for example) to a fairly complex logical form (Harabagiu et al., 2001; El- worthy, 2000, for example). Once the question is analysed, a document preselection module iden- tifies the documents that are most likely to contain the answer. This module typically uses informa- tion retrieval techniques that rely on bag-of-words approaches and statistical information (Voorhees, 2001). A filtering module examines the resulting documents and selects or rewards the named enti- ties that are compatible with the question type. A scoring module then performs a more intensive analysis and ranks the preselected named entities (an possibly surrounding text) according to their likelihood to contain the answer. The scoring sys- tem relies on the output given by the question analysis module and possibly the images of the preselected documents that were created …

Content truncated to 500 words