FARMI: A FrAmework for Recording Multi-Modal Interactions

FARMI: A FrAmework for Recording Multi-Modal Interactions Patrik Jonell1, Mattias Bystedt1, Per Fallgren1 Dimosthenis Kontogiorgos1, José Lopes1, Zofia Malisz1 Samuel Mascarenhas 2, Catharine Oertel1, Eran Raveh3, Todd Shore1 1 KTH Speech, Music and Hearing, Stockholm, Sweden 2 GAIPS INESC-ID, Lisbon, Portugal 3 Multimodal Computing and Interaction, Saarland University, Germany pjjonell@kth.se Abstract In this paper we present (1) a processing architecture used to collect multi-modal sensor data, both for corpora collection and real-time processing, (2) an open-source implementation thereof and (3) a use-case where we deploy the architecture in a multi-party deception game, featuring six human players and one robot. The architecture is agnostic to the choice of hardware (e.g. microphones, cameras, etc.) and programming languages, although our implementation is mostly written in Python. In our use-case, different methods of capturing verbal and non-verbal cues from the participants were used. These were processed in real-time and used to inform the robot about the participants’ deceptive behaviour. The framework is of particular interest for researchers who are interested in the collection of multi-party, richly recorded corpora and the design of conversational systems. Moreover for researchers who are interested in human-robot interaction the available modules offer the possibility to easily create both autonomous and wizard-of-Oz interactions. Keywords: multisensory processing, human-robot interaction, multimodal interaction 1. Introduction Recording group interactions is an essential step towards a better understanding of the mechanisms of multi-party in- teraction both between humans and between humans and robots. In such interactions, humans communicate via a number of different channels. Expressions in any modal- ity, as well as the interactions between modalities and par- ticipants, need to be reliably captured to enable a proper analysis of how humans communicate in group settings. Capturing multi-modal and multi-party interaction is still a tremendous challenge in terms of hardware and software development. Recordings of this kind generate very large amounts of data. Ensuring synchronisation between sensor, audio and video streams is crucial for reliable analysis. When it comes to recording multi-modal datasets or imple- menting multi-modal dialogue systems, it is difficult to find a framework that has been widely adopted by the commu- nity. Given that there are a lot of constraints imposed by the hardware, often the only option left is to develop a frame- work around that same hardware. In addition, such cus- tom frameworks frequently lack detailed documentation. There were previous attempts to develop frameworks that support multi-modal dialogue systems, for instance (Bohus and Horvitz, 2009) and IrisTK (Skantze and Al Moubayed, 2012). IrisTK enables the recording of multi-modal data collections, however, data stream synchronisation is not fully solved. Mint.tools (Kousidis et al., 2013) tackle syn- chronisation problems and perform recordings in an anno- tation friendly manner, however this framework does not close the loop into a dialogue system and is not, to our knowledge, publicly available. An artificial agent needs to have additional methods of per- ceiving human behaviour in order to be context aware in human-robot interaction (Turk, 2014). Apart from natu- ral language understanding, several shallow input modali- …

Content truncated to 500 words