Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1308–1319 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1120 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1308–1319 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1120 Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems Satoshi Akasaki∗ The University of Tokyo akasaki@tkl.iis.u-tokyo.ac.jp Nobuhiro Kaji Yahoo Japan Corporation nkaji@yahoo-corp.jp Abstract Recently emerged intelligent assistants on smartphones and home electronics (e.g., Siri and Alexa) can be seen as novel hybrids of domain-specific task- oriented spoken dialogue systems and open-domain non-task-oriented ones. To realize such hybrid dialogue systems, this paper investigates determining whether or not a user is going to have a chat with the system. To address the lack of bench- mark datasets for this task, we construct a new dataset consisting of 15, 160 utter- ances collected from the real log data of a commercial intelligent assistant (and will release the dataset to facilitate future re- search activity). In addition, we investi- gate using tweets and Web search queries for handling open-domain user utterances, which characterize the task of chat de- tection. Experiments demonstrated that, while simple supervised methods are ef- fective, the use of the tweets and search queries further improves the F1-score from 86.21 to 87.53. 1 Introduction 1.1 Chat detection Conventional studies on spoken dialogue systems (SDS) have investigated either domain-specific task-oriented SDS1 (Williams and Young, 2007) or open-domain non-task-oriented SDS (a.k.a., chatbots or chat-oriented SDS) (Wallace, 2009). The former offers convenience by helping users complete tasks in specific domains, while the latter tion.∗Work done during internship at Yahoo Japan Corpora- 1They can be classified as single-domain or multi-domain task-oriented SDS. offers entertainment through open-ended chatting (or smalltalk) with users. Although the function- alities offered by the two types of SDS are com- plementary to each other, little practical effort has been made to combine them. This unfortunately has limited the potential of SDS. This situation is now being changed by the emergence of voice-activated intelligent assistants on smartphones and home electronics (e.g., Siri2 and Alexa3). These intelligent assistants typically perform various tasks (e.g., Web search, weather checking, and alarm setting) while being able to have chats with users. They can be seen as a novel hybrid of multi-domain task-oriented SDS and open-domain non-task-oriented SDS. To realize such hybrid SDS, we have to deter- mine whether or not a user is going to have a chat with the system. For example, if a user says “What is your hobby?” it is considered that she is going to have a chat with the system. On the other hand, if she says “Set an alarm at 8 o’clock,” she is prob- ably trying to operate her smartphone. We refer to this task as chat detection and treat it as a binary classification problem. Chat detection has not been explored enough in past studies. This is primarily …

Content truncated to 500 words