A Multi-party Multi-modal Dataset for Focus of Visual Attention in Human-human and Human-robot Interaction

A Multi-party Multi-modal Dataset for Focus of Visual Attention in Human-human and Human-robot Interaction Kalin Stefanov and Jonas Beskow KTH Royal Institute of Technology Stockholm, Sweden kalins@kth.se and beskow@kth.se Abstract This papers describes a data collection setup and a newly recorded dataset. The main purpose of this dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The dataset contains two parts - 6 sessions with duration of approximately 3 hours and 9 sessions with duration of approximately 4.5 hours. Both parts of the dataset are rich in modalities and recorded data streams - they include the streams of three Kinect v2 devices (color, depth, infrared, body and face data), three high quality audio streams, three high resolution GoPro video streams, touch data for the task-based interactions and the system state of the robot. In addition, the second part of the dataset introduces the data streams from three Tobii Pro Glasses 2 eye trackers. The language of all interactions is English and all data streams are spatially and temporally aligned. Keywords: dataset, focus of visual attention, human-robot interaction, human-human interaction 1. Introduction During recent years, we have witnessed the start of a revolution in personal robotics that is rapidly starting to become part of our everyday lives. The promise and potential of these systems is far-reaching; from general purpose co-worker robots that operate and collaborate with humans side-by-side, via household and service robots that are able to carry out or assist in daily chores to robotic tutors in schools that interact with humans in and around a shared environment. All of these scenarios require systems that are able to recognize and convey attention to objects in the environment. Human face-to-face interaction involves sophisticated mechanisms for conveying inten- tions, establishing common ground and acknowledging information transfer, often based on a combination of spoken and visual (nonverbal) signals. In order to be able to understand and model these processes and leverage their power in human-robot interaction, we need comprehensive multi-modal data collected in relevant settings. In this paper, we present a highly multi-modal interaction dataset that was recorded for this purpose. It involves multi-party human-robot interaction based around objects (virtual cards on a touch table), multi-party human-human interaction based around the same objects, as well as free multi-party human-human interaction without objects. The primary purpose of the dataset is to serve as a source for modeling visual attention patterns for robots interacting with hu- mans, but the richness of the dataset also makes it useful for other studies on the dynamics of multi-party interaction. So why a new dataset when there is already a wealth of multi-party interaction recordings that have been made? There are three main reasons for this, • We want data for interaction in multiple settings, in particular with/without object …

Content truncated to 500 words