What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 22–32, Prague, June 2007. c⃝2007 Association for Computational Linguistics What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA Mengqiu Wang and Noah A. Smith and Teruko Mitamura Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 USA {mengqiu,nasmith,teruko}@cs.cmu.edu Abstract This paper presents a syntax-driven ap- proach to question answering, specifically the answer-sentence selection problem for short-answer questions. Rather than us- ing syntactic features to augment exist- ing statistical classifiers (as in previous work), we build on the idea that ques- tions and their (correct) answers relate to each other via loose but predictable syntac- tic transformations. We propose a prob- abilistic quasi-synchronous grammar, in- spired by one proposed for machine trans- lation (D. Smith and Eisner, 2006), and pa- rameterized by mixtures of a robust non- lexical syntax/alignment model with a(n optional) lexical-semantics-driven log-linear model. Our model learns soft alignments as a hidden variable in discriminative training. Experimental results using the TREC dataset are shown to significantly outperform strong state-of-the-art baselines. 1 Introduction and Motivation Open-domain question answering (QA) is a widely- studied and fast-growing research problem. State- of-the-art QA systems are extremely complex. They usually take the form of a pipeline architecture, chaining together modules that perform tasks such as answer type analysis (identifying whether the correct answer will be a person, location, date, etc.), document retrieval, answer candidate extrac- tion, and answer reranking. This architecture is so predominant that each task listed above has evolved into its own sub-field and is often studied and evalu- ated independently (Shima et al., 2006). At a high level, the QA task boils down to only two essential steps (Echihabi and Marcu, 2003). The first step, retrieval, narrows down the search space from a corpus of millions of documents to a fo- cused set of maybe a few hundred using an IR en- gine, where efficiency and recall are the main fo- cus. The second step, selection, assesses each can- didate answer string proposed by the first step, and finds the one that is most likely to be an answer to the given question. The granularity of the tar- get answer string varies depending on the type of the question. For example, answers to factoid ques- tions (e.g., Who, When, Where) are usually single words or short phrases, while definitional questions and other more complex question types (e.g., How, Why) look for sentences or short passages. In this work, we fix the granularity of an answer to a single sentence. Earlier work on answer selection relies only on the surface-level text information. Two approaches are most common: surface pattern matching, and similarity measures on the question and answer, rep- resented as bags of words. In the former, pat- terns for a certain answer type are either crafted manually (Soubbotin and Soubbotin, 2001) or ac- quired from training examples automatically (Itty- cheriah et al., 2001; Ravichandran et al., 2003; …

Content truncated to 500 words