AppDialogue: Multi-App Dialogues for Intelligent Assistants

AppDialogue: Multi-App Dialogues for Intelligent Assistants Ming Sun∗ Yun-Nung Chen∗ Zhenhao Hua† Yulian Tamres-Rudnicky‡ Arnab Dash⋆ Alexander I. Rudnicky∗ ∗†⋆School of Computer Science, Carnegie Mellon University †Pinterest ⋆Ernst & Young ∗{mings,yvchen,air}@cs.cmu.edu †‡⋆{troyhua0426,yulie355,arnabdash92}@gmail.com Abstract Users will interact with an individual app on smart devices (e.g., phone, TV, car) to fulfill a specific goal (e.g. find a photographer), but users may also pursue more complex tasks that will span multiple domains and apps (e.g. plan a wedding ceremony). Planning and executing such multi-app tasks are typically managed by users, considering the required global context awareness. To investigate how users arrange domains/apps to fulfill complex tasks in their daily life, we conducted a user study on 14 participants to collect such data from their Android smart phones. This document 1) summarizes the techniques used in the data collection and 2) provides a brief statistical description of the data. This data guilds the future direction for researchers in the fields of conversational agent and personal assistant, etc. This data is available at http://AppDialogue.com. Keywords: spoken dialog system; multi-domain interaction; intelligent agent. 1. Introduction Smart devices, such as smart phones or TVs, allow users to achieve their goals (intentions) through verbal and non- verbal communication. The intention sometimes can be ful- filled in one single domain (i.e., an app). However, it is possible to span multiple domains and requires information coordination among these domains. A human user, with the global context at hand, can well-organize the functionality provided by individual apps and coordinate information ef- ficiently. In other word, user can mentally create his own virtual app on top of existing ones. On the other hand, al- though intelligent agents can be configured by developers to passively support (limited) types of cross-domain inter- actions, they are not capable of actively managing apps to satisfy a user’s potentially complex intentions, because they do not consider the repeated execution of activities in pur- suit of user intentions. Currently, most human-machine interactions are carried out via touch-screen. Although the recognizable gestures have been expanded during the past decade (Harrison et al., 2014), interactive expressions are still restricted due to the limit of gestures and displays. This may affect usability for certain populations, such as older users or users with visual disabilities. By contrast, spoken language can effec- tively convey the user’s high-level and complex intentions to a device (e.g., Apple Siri, Amazon Alexa, Google Now and Microsoft Cortana). However speech presents the chal- lenges about 1) understanding both at the level of individual apps and at the level of tasks that span apps; and 2) commu- nicating a task-level functionality between user and agent. This presented data may function as a testbed for smart de- vices to address aforementioned challenges. In order to investigate how human users arrange apps to- gether in their daily life, and also to understand how they would interact with intelligent assistants via speech instead, we designed this user study. We describe data collection and involved techniques in Section 2. Then we provide statistics …

Content truncated to 500 words