Is Question Answering Better than Information Retrieval? Towards a Task-Based Evaluation Framework for Question Series

Proceedings of NAACL HLT 2007, pages 212–219, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Is Question Answering Better Than Information Retrieval? Towards a Task-Based Evaluation Framework for Question Series Jimmy Lin College of Information Studies Department of Computer Science Institute for Advanced Computer Studies University of Maryland College Park, MD 20742, USA jimmylin@umd.edu Abstract This paper introduces a novel evaluation framework for question series and em- ploys it to explore the effectiveness of QA and IR systems at addressing users’ infor- mation needs. The framework is based on the notion of recall curves, which char- acterize the amount of relevant informa- tion contained within a fixed-length text segment. Although it is widely assumed that QA technology provides more effi- cient access to information than IR sys- tems, our experiments show that a simple IR baseline is quite competitive. These re- sults help us better understand the role of NLP technology in QA systems and sug- gest directions for future research. 1 Introduction The emergence of question answering (QA) has been driven to a large extent by its intuitive appeal. Instead of “hits”, QA technology promises to de- liver “answers”, obviating the user from the tedious task of sorting through lists of potentially-relevant documents. The success of factoid QA systems, particularly in the NIST-sponsored TREC evalua- tions (Voorhees, 2003), has reinforced the percep- tion about the superiority of QA systems over tradi- tional IR engines. However, is QA really better than IR? This work challenges existing assumptions and critically exam- ines this question, starting with the development of a novel evaluation framework that better models user tasks and preferences. The framework is then ap- plied to compare top TREC QA systems against an off-the-shelf IR engine. Surprisingly, experiments show that the IR baseline is quite competitive. These results help us better understand the added value of NLP technology in QA systems, and are also useful in guiding future research. 2 Evolution of QA Evaluation Although most question answering systems rely on information retrieval technology, there has always been the understanding that NLP provides signifi- cant added value beyond simple IR. Even the earli- est open-domain factoid QA systems, which can be traced back to the late nineties (Voorhees and Tice, 1999), demonstrated the importance and impact of linguistic processing. Today’s top systems deploy a wide range of advanced NLP technology and can answer over three quarters of factoid questions in an open domain (Voorhees, 2003). However, present QA evaluation methodology does not take into ac- count two developments, discussed below. First, despite trends to the contrary in TREC eval- uations, users don’t actually like or want exact an- swers. Most question answering systems are de- signed to pinpoint the exact named entity (person, date, organization, etc.) that answers a particular question—and the development of such technology has been encouraged by the setup of the TREC QA tracks. However, a study by Lin et al. (2003) shows that users actually prefer answers embedded within some sort of context, e.g., the …

Content truncated to 500 words