Detecting Bot-Answerable Questions in Ubuntu Chat

International Joint Conference on Natural Language Processing, pages 747–752, Nagoya, Japan, 14-18 October 2013. Detecting Bot-Answerable Questions in Ubuntu Chat David C. Uthus NRC/NRL Postdoctoral Fellow Washington, DC 20375 duthus@google.com David W. Aha Navy Center for Applied Research in AI Naval Research Laboratory (Code 5514) Washington, DC 20375 david.aha@nrl.navy.mil Abstract Ubuntu’s Internet Relay Chat technical support channel has bots that output specific messages in response to com- mand words from other channel users. These messages can be used to answer frequently-asked questions instead of re- quiring an expert to (repeatedly) type a lengthy reply. We describe an approach to automatically distinguish bot-answerable questions, which would mitigate this prob- lem. To the best of our knowledge, this is the first work on investigating question an- swering in a multiparticipant chat domain. Our results indicate that for some types of questions, supervised learning algorithms perform well on this task and, in addition, that character n-grams are a better repre- sentation than traditional bag-of-words for this task and domain. 1 Introduction Ubuntu (a Linux-based operating system) main- tains multiple Internet Relay Chat (IRC) chan- nels for technical support. Some of these chan- nels contain bots, which are automated agents pre-programmed to perform certain tasks. One of the bots can output pre-written messages, called factoids, in response to command words. For example, if a user types “!flash”, then the bot would output “To install Flash see https://help.ubuntu....mats/Flash - See also !Re- stricted and !Gnash”. These factoids are used to answer common questions, enforce channel guide- lines, direct non-English speakers (in their na- tive tongue) to Ubuntu’s foreign language support channels, and query Ubuntu’s repository of pack- ages. invoked.While useful, this bot must be manually Automating the bot to self-detect and answer questions that it can answer could reduce the workload for knowledgeable experts trying to help other users. This is applicable to not only Ubuntu’s technical support channels, but to other IRC channels providing technical support (e.g., Debian’s support channels) and to channels that use similar bots (e.g., Eggdrop and Infobot) for other purposes. We describe initial steps on a self-invoking bot. We begin by investigating the multi-classification task of which questions are bot-answerable ques- tions (BAQ) and which are human-answerable questions (HAQ), which requires a human to an- swer due to the question’s complexity. We imple- mented a baseline non-learning approach and su- pervised support vector machines (SVM) and k- nearest neighbor (k-NN) algorithms. Our results show that a bot can answer with confidence some types of questions, especially those directing users to more appropriate channels for help on certain topics. Our contributions are as follows: • Problem: We identified a real-world problem that has not been investigated, despite bots having been around for years on IRC chan- nels. • Data: We created an annotated multipartici- pant chat corpus that is publicly available. • Empirical study: We report on our investi- gation of applying supervised learning algo- rithms and leveraging different feature rep- resentations, whose results will be used as a foundation for …

Content truncated to 500 words