Crowdsourcing for (almost) Real-time Question Answering

Proceedings of 2016 NAACL Human-Computer Question Answering Workshop, pages 8–14, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Crowdsourcing for (almost) Real-time Question Answering Denis Savenkov Emory University dsavenk@emory.edu Scott Weitzner Emory University sweitzn@emory.edu Eugene Agichtein Emory University eugene@mathcs.emory.edu Abstract Modern search engines have made dramatic progress in the answering of many user’s ques- tions about facts, such as those that might be retrieved or directly inferred from a knowl- edge base. However, many other questions that real users ask are more complex, such as asking for opinions or advice for a particular situation, and are still largely beyond the com- petence of the computer systems. As conver- sational agents become more popular, QA sys- tems are increasingly expected to handle such complex questions, and to do so in (nearly) real-time, as the searcher is unlikely to wait longer than a minute or two for an answer. One way to overcome some of the challenges in complex question answering is crowdsourc- ing. We explore two ways crowdsourcing can assist a question answering system that op- erates in (near) real time: by providing an- swer validation, which could be used to filter or re-rank the candidate answers, and by cre- ating the answer candidates directly. Specifi- cally, we focus on understanding the effects of time restrictions in the near real-time QA set- ting. Our experiments show that even within a one minute time limit, crowd workers can produce reliable ratings for up to three answer candidates, and generate answers that are bet- ter than an average automated system from the LiveQA 2015 shared task. Our findings can be useful for developing hybrid human-computer systems for automatic question answering and conversational agents. 1 Introduction It has long been a dream to communicate with a computer as one might with another human being using natural language speech and text. Nowadays, we are coming closer to this dream, as natural lan- guage interfaces become increasingly popular. Our phones are already reasonably good at recognizing speech, and personal assistants, such as Apple Siri, Google Now, Microsoft Cortana, Amazon Alexa, etc., help us with everyday tasks and answer some of our questions. Chat bots are arguably considered “the next big thing”, and a number of startups devel- oping this kind of technology has emerged in Silicon Valley and around the world1. Question answering is one of the major compo- nents of such personal assistants. Existing tech- niques already allow users to get direct answers to their factoid questions. However, there is still a large number of more complex questions, such as advice or accepted general opinions, for which users have to dig into the “10 blue links” and extract or syn- thesize answers from information buried within the retrieved documents. To cater to these informational needs, community question answering (CQA) sites emerged, such as Yahoo! Answers and Stack Ex- change. These sites provide a popular way to con- nect information seekers with answerers. Unfortu- nately, it can take minutes or hours, and sometimes …

Content truncated to 500 words