Using Syntactic and Semantic Structural Kernels for Classifying Definition Questions in Jeopardy!

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 712–724, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Using Syntactic and Semantic Structural Kernels for Classifying Definition Questions in Jeopardy! Alessandro Moschitti† Jennifer Chu-Carroll‡ Siddharth Patwardhan‡ James Fan‡ Giuseppe Riccardi† †Department of Information Engineering and Computer Science University of Trento, 38123 Povo (TN), Italy {moschitti,riccardi}@disi.unitn.it ‡IBM T.J. Watson Research Center P.O. Box 704, Yorktown Heights, NY 10598, U.S.A. {jencc,siddharth,fanj}@us.ibm.com Abstract The last decade has seen many interesting ap- plications of Question Answering (QA) tech- nology. The Jeopardy! quiz show is certainly one of the most fascinating, from the view- points of both its broad domain and the com- plexity of its language. In this paper, we study kernel methods applied to syntactic/semantic structures for accurate classification of Jeop- ardy! definition questions. Our extensive em- pirical analysis shows that our classification models largely improve on classifiers based on word-language models. Such classifiers are also used in the state-of-the-art QA pipeline constituting Watson, the IBM Jeopardy! sys- tem. Our experiments measuring their impact on Watson show enhancements in QA accu- racy and a consequent increase in the amount of money earned in game-based evaluation. 1 Introduction Question Answering (QA) is an important research area of Information Retrieval applications, which re- quires the use of core NLP capabilities, such as syn- tactic and semantic processing for a more effective user experience. While the development of most existing QA systems are driven by organized eval- uation efforts such as TREC (Voorhees and Dang, 2006), CLEF (Giampiccolo et al., 2007), and NT- CIR (Sasaki et al., 2007), there exist efforts that leverage data from popular quiz shows, such as Who Wants to be a Millionaire (Clarke et al., 2001; Lam et al., 2003) and Jeopardy! (Ferrucci et al., 2010), to demonstrate the generality of the technology. Jeopardy! is a popular quiz show in the US which has been on the air for 27 years. In each game, three contestants compete for the opportunity to answer 60 questions in 12 categories of 5 questions each. Jeopardy! questions cover an incredibly broad do- main, from science, literature, history, to popular culture. We are drawn to Jeopardy! as a test bed for open-domain QA technology due to its broad do- main, complex language, as well as the emphasis on accuracy, confidence, and speed during game play. While the vast majority of Jeopardy! questions are factoid questions, we find several other types of questions in the Jeopardy! data, which can ben- efit from specialized processing in the QA system. The additional processing in these questions com- plements that of the factoid questions to achieve im- proved overall QA performance. Among the various types of questions handled by the system are defini- tion questions shown in the examples below: (1) GON TOMORROW: It can be the basket below a hot-air balloon or a flat-bottomed boat used on a canal (answer: gondola); (2) I LOVE YOU, “MIN”: Overbearing (an- swer: domineering); (3) INVEST: From …

Content truncated to 500 words