Prediction of Prospective User Engagement with Intelligent Assistants

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1203–1212, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Prediction of Prospective User Engagement with Intelligent Assistants Shumpei Sano, Nobuhiro Kaji, and Manabu Sassano Yahoo Japan Corporation 9-7-1 Akasaka, Minato-ku, Tokyo 107-6211, Japan {shsano, nkaji, msassano}@yahoo-corp.jp Abstract Intelligent assistants on mobile devices, such as Siri, have recently gained con- siderable attention as novel applications of dialogue technologies. A tremendous amount of real users of intelligent assis- tants provide us with an opportunity to ex- plore a novel task of predicting whether users will continually use their intelligent assistants in the future. We developed pre- diction models of prospective user engage- ment by using large-scale user logs ob- tained from a commercial intelligent as- sistant. Experiments demonstrated that our models can predict prospective user engagement reasonably well, and outper- forms a strong baseline that makes predic- tion based past utterance frequency. 1 Introduction Intelligent assistants on mobile devices, such as Siri,1 have recently gained considerable atten- tion as novel applications of dialogue technologies (Jiang et al., 2015). They receive instructions from users via voice control to execute a wide range of tasks (e.g., searching the Web, setting alarms, making phone calls, and so on). Some are able to even chat or play games with users (Kobayashi et al., 2015). Intelligent assistants possess a unique character- istic as an object of dialogue study. Popular intel- ligent assistants have thousands or even millions of real users, thanks to the prevalence of mobile devices. Some of those users continually use in- telligent assistants for a long period of time, while others stop using them after a few trials. Such user behaviors are rarely observed in conventional ex- perimental environments, where dialogue systems 1http://www.apple.com/ios/siri have only a small number of experimental partici- pants who almost always continue to use the sys- tems for the whole duration of the experiment. This paper explores a novel task of predicting whether a user will continue to use intelligent as- sistants in the future (This task is referred to as prospective user engagement prediction and its definition is given in Section 3). We attempt to de- velop such a prediction model, which would con- tribute to enhancing intelligent assistants in many ways. For example, if users who are likely to stop using systems can be identified, intelligent assis- tants can take actions to gain or maintain their in- terest (e.g., by sending push notifications). This task is related to, but is significantly differ- ent from, user engagement detection, which has been extensively explored in prior dialogue stud- ies (Wang and Hirschberg, 2011; Forbes-Riley et al., 2012; Forbes-Riley and Litman, 2013; Oertel and Salvi, 2013). The prior studies attempt to pre- dict how strongly users are currently engaged in dialogues with systems. On the other hand, the goal of this study is to predict how strongly users will be engaged with intelligent assistants in the future. The largest difference lies in whether the prediction target …

Content truncated to 500 words