How to Address Smart Homes with a Social Robot? A Multi-modal Corpus of User Interactions with an Intelligent Environment

How to Address Smart Homes with a Social Robot? A Multi-modal Corpus of User Interactions with an Intelligent Environment Patrick Holthaus, Christian Leichsenring, Jasmin Bernotat, Viktor Richter, Marian Pohling, Birte Carlmeyer, Norman K¨oster, Sebastian Meyer zu Borgsen, Ren´e Zorn, Birte Schiffhauer, Kai Frederic Engelmann, Florian Lier, Simon Schulz, Philipp Cimiano, Friederike Eyssel, Thomas Hermann, Franz Kummert, David Schlangen, Sven Wachsmuth, Petra Wagner, Britta Wrede, Sebastian Wrede Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Inspiration 1, 33619 Bielefeld, Germany patrick.holthaus@uni-bielefeld.de Abstract In order to explore intuitive verbal and non-verbal interfaces in smart environments we recorded user interactions with an intelligent apartment. Besides offering various interactive capabilities itself, the apartment is also inhabited by a social robot that is available as a humanoid interface. This paper presents a multi-modal corpus that contains goal-directed actions of naive users in attempts to solve a number of predefined tasks. Alongside audio and video recordings, our data-set consists of large amount of temporally aligned sensory data and system behavior provided by the environment and its interactive components. Non-verbal system responses such as changes in light or display contents, as well as robot and apartment utterances and gestures serve as a rich basis for later in-depth analysis. Manual annotations provide further information about meta data like the current course of study and user be- havior including the incorporated modality, all literal utterances, language features, emotional expressions, foci of attention, and addressees. Keywords: interaction corpus, smart home, social robot 1. Introduction With smart home technologies becoming increasingly widespread, for example in elderly care (Morris et al., 2013; Hendrich et al., 2014; Cavallo et al., 2014; Auberg´e et al., 2014), new opportunities for the collection of interaction data arise. Such environments thereby promise a dense web of functionalities and services that are so seamlessly inte- grated into living spaces that artifacts to control, query or program them all but vanish. In fact, if every component pos- sessed its own interface, the environment would be utterly cluttered and lose its familiarity and calmness. The question arises how to bridge the interface gap and how to control these systems in an easy, intuitive, or – better – natural way. One approach to create novel device interaction strategies is to start bottom-up – i. e. from observing how inexperienced users would do it if they could do whatever they like to initiate a desired function (e. g. Valdes et al. (2014)). In this work, we present a multi-modal corpus on goal attain- ment strategies in a number of experimental situations, care- fully selected to avoid a specific bias regarding the modal- ities to use for interactions. The data accordingly contains free-form interaction sequences with the environment that are not restricted in the way given tasks are performed by the participants. To this end, individual users trigger a set of specific de- vice actions and responses in our cognitive service-robotics apartment (CSRA)1, a smart apartment embedded in our research building that offers services both from a variety of smart ambient components …

Content truncated to 500 words