Experiments with Interactive Question-Answering

Proceedings of the 43rd Annual Meeting of the ACL, pages 205–214, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Experiments with Interactive Question-Answering Sanda Harabagiu, Andrew Hickl, John Lehmann, and Dan Moldovan Language Computer Corporation Richardson, Texas USA sanda@languagecomputer.com Abstract This paper describes a novel framework for interactive question-answering (Q/A) based on predictive questioning. Gen- erated off-line from topic representations of complex scenarios, predictive ques- tions represent requests for information that capture the most salient (and diverse) aspects of a topic. We present experimen- tal results from large user studies (featur- ing a fully-implemented interactive Q/A system named FERRET) that demonstrates that surprising performance is achieved by integrating predictive questions into the context of a Q/A dialogue. 1 Introduction In this paper, we propose a new architecture for interactive question-answering based on predictive questioning. We present experimental results from a currently-implemented interactive Q/A system, named FERRET, that demonstrates that surprising performance is achieved by integrating sources of topic information into the context of a Q/A dialogue. In interactive Q/A, professional users engage in extended dialogues with automatic Q/A systems in order to obtain information relevant to a complex scenario. Unlike Q/A in isolation, where the per- formance of a system is evaluated in terms of how well answers returned by a system meet the specific information requirements of a single question, the performance of interactive Q/A systems have tradi- tionally been evaluated by analyzing aspects of the dialogue as a whole. Q/A dialogues have been evalu- ated in terms of (1) efficiency, defined as the number of questions that the user must pose to find particu- lar information, (2) effectiveness, defined by the rel- evance of the answers returned, (3) user satisfaction. In order to maximize performance in these three areas, interactive Q/A systems need a predictive di- alogue architecture that enables them to propose re- lated questions about the relevant information that could be returned to a user, given a domain of inter- est. We argue that interactive Q/A systems depend on three factors: (1) the effective representation of the topic of a dialogue, (2) the dynamic recognition of the structure of the dialogue, and (3) the ability to return relevant answers to a particular question. In this paper, we describe results from experi- ments we conducted with our own interactive Q/A system, FERRET, under the auspices of the ARDA AQUAINT1 program, involving 8 different dialogue scenarios and more than 30 users. The results pre- sented here illustrate the role of predictive question- ing in enhancing the performance of Q/A interac- tions. In the remainder of this paper, we describe a new architecture for interactive Q/A. Section 2 presents the functionality of several of FERRET’s modules and describes the NLP techniques it relies upon. In Section 3, we present one of the dialogue scenar- ios and the topic representations we have employed. Section 4 highlights the management of the inter- action between the user and FERRET, while Sec- tion 5 presents the results of evaluating our proposed 1AQUAINT …

Content truncated to 500 words