Multimodal Resources for Human-Robot Communication Modelling

Multimodal Resources for Human-Robot Communication Modelling Stavroula–Evita Fotinea, Eleni Efthimiou, Maria Koutsombogera, Athanasia–Lida Dimou, Theodore Goulas, Kyriaki Vasilaki Institute for Language and Speech Processing - ILSP/ATHENA RC Artemidos 6 & Epidavrou, Maroussi, 15125 Athens, Greece {evita, eleni_e, mkouts, eleni_e, ndimou, tgoulas}@ilsp.gr, kiki-vas@hotmail.com Abstract This paper reports on work related to the modelling of Human-Robot Communication on the basis of multimodal and multisensory human behaviour analysis. A primary focus in this framework of analysis is the definition of semantics of human actions in interaction, their capture and their representation in terms of behavioural patterns that, in turn, feed a multimodal human-robot communication system. Semantic analysis encompasses both oral and sign languages, as well as both verbal and non-verbal communicative signals to achieve an effective, natural interaction between elderly users with slight walking and cognitive inability and an assistive robotic platform. Keywords: multisensory data acquisition, multimodal semantics, multimodal annotation scheme, sign language, multimodal HRI model extraction, multimodal human-robot communication, natural HRI 1. Introduction Cognitive and robotic architectures may be able to provide advanced interactive capabilities with humans and influence the usability and functionality of a resulting system, contributing to its quality of services. Such systems may not only integrate multiple advanced cognitive abilities, but also employ methods that are extendible to various other robotic and non-robotic applications required in assisting humans with mobility disabilities. In this work, the driving concept envisions cognitive robotic assistants that act (a) proactively by realizing an autonomous and context-specific monitoring of human activities and by subsequently reasoning on meaningful user behavioural patterns, as well as (b) adaptively and interactively, by analyzing multi-sensory and physiological signals related to gait and postural stability, and by performing adaptive compliance control for optimal physical support and active fall prevention. A primary focus in this work has been the analysis of available data and the definition of the semantics of human actions in interaction between humans. At a second stage, their capture and representation in terms of behavioural patterns have been exploited to feed a multimodal human-robot communication system. Semantic analysis, in this sense, encompasses both oral and sign languages, as well as both verbal and non-verbal communicative signals. Towards defining the system’s multimodal HRI model, we discuss the acquisition of the multimodal sensory corpus which served as a primary source of data retrieval, analysis and testing of the developed mobility assistive robot prototypes. We further focus on the definition of the representation scheme of audio-gestural signals which provided the ground truth dataset for the training and evaluation of visual and audio recognition algorithms, and ensured the usability of the acquired dataset in defining the adopted multimodal human-robot communication model. Acquisition, annotation and analysis of the here discussed multimodal-multisensory dataset as well as work on definition of the HRI communication model based on this specific dataset have taken place in the framework of the MOBOT project 1 , which aims at developing intelligent active mobility assistance robots for indoor environments that provide user-centred, context-adaptive and natural support. Specifically, a multimodal action recognition …

Content truncated to 500 words