Watson Discovery Advisor: Question-answering in an industrial setting

Proceedings of 2016 NAACL Human-Computer Question Answering Workshop, pages 1–7, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Watson Discovery Advisor: Question-answering in an industrial setting Charles Beller and Graham Katz and Allen Ginsberg and Chris Phipps cebeller@us.ibm.com, egkatz@us.ibm.com, abginsbe@us.ibm.com, phippsc@us.ibm.com Sean Bethard and Paul Chase and Elinna Shek and Kristen Summers slbethar@us.ibm.com, pjchase@us.ibm.com, eshek@us.ibm.com, kmsummer@us.ibm.com IBM Watson Group Chantilly, VA, United States Abstract This work discusses a mix of challenges aris- ing from Watson Discovery Advisor (WDA), an industrial strength descendant of the Wat- son Jeopardy! Question Answering system currently used in production in industry set- tings. Typical challenges include generation of appropriate training questions, adaptation to new industry domains, and iterative im- provement of the system through manual error analyses. 1 Introduction Question answering has been a focus for research in computational linguistics and natural language pro- cessing since early efforts in the 1960s (Simmons, 1970). Systems such as BASEBALL (Green Jr. et al., 1961) focused on specific, circumscribed do- mains, e.g. questions about baseball games. As computing power and knowledge resources devel- oped, answering questions over a broad open do- main became a focus of engineers and researchers in government, industry, academic settings (Voorhees, 1999; Wang, 2006). Various approaches to ques- tion answering have exploited logical forms and logic provers (Moldovan et al., 2003a), relationship matching over structured knowledge bases (Yao and Van Durme, 2014), and the wisdom of the masses via social media analysis (Bian et al., 2008) or crowd-sourcing (Boyd-Graber et al., 2012). Open-Domain question answering received a wave of renewed attention when IBM’s Watson sys- tem successfully competed with human champions in the Jeopardy! challenge. That system was devel- oped into the Watson Discovery Advisor IBM prod- uct, which we describe below. 2 Watson Discovery Advisor The Watson Discovery Advisor product (WDA) is a direct descendant of the Watson Jeopardy! sys- tem described in (Ferrucci et al., 2010; Ferrucci, 2012). The core processing pipelines are built on the Apache UIMA architecture (Ferrucci and Lally, 2004) and consist of a question-analysis phase (Lally et al., 2012; Kalyanpur et al., 2012), a doc- ument and passage retrieval phase, an answer gen- eration phase (Chu-Carroll et al., 2012), an answer scoring phase, and an answer merging and ranking phase (Gondek et al., 2012). There are several important differences between the Jeopardy! task and the question answering task performed by WDA. At the most superficial level, the Jeopardy! task provides an input consisting of a declarative clue, and expects an interrogative ques- tion, i.e. it’s an Answer-Questioning task. In reality though, this tends to be a simple syntactic transfor- mation. A more important difference is that Jeop- ardy! clues are always accompanied by category information. In some cases, this category level in- formation is crucial, giving explicit parameters that restrict the range of answers to be considered. Another difference has to do with the nature of the input questions. Jeopardy! includes a range of puzzle-type questions that required special logic (Prager et al., 2012), these …

Content truncated to 500 words