The OFAI Multi-Modal Task Description Corpus

The OFAI Multimodal Task Description Corpus Stephanie Gross, Brigitte Krenn Austrian Research Institute for Artificial Intelligence Freyung 6, 1010 Vienna, Austria {stephanie.gross, brigitte.krenn}@ofai.at Abstract The OFAI Multimodal Task Description Corpus (OFAI-MMTD Corpus) is a collection of dyadic teacher-learner (human-human and human-robot) interactions. The corpus is multimodal and tracks the communication signals exchanged between interlocutors in task-oriented scenarios including speech, gaze and gestures. The focus of interest lies on the communicative signals conveyed by the teacher and which objects are salient at which time. Data are collected from four different task description setups which involve spatial utterances, navigation instructions and more complex descriptions of joint tasks. Keywords: multimodal task description, open world reference resolution, multimodal human-robot interaction 1. Introduction Future robots are expected to be present in people’s homes, collaborate with and support their human users in various everyday activities and tasks including mobility, manipula- tion, personal care, fetch and carry support in the house- hold, and so forth. Bringing robots into real-world and thus open environments requires, amongst many other aspects of research and technological development, the creation of artificial learners that can learn from being exposed to task descriptions given by a human tutor. These task descrip- tions are multimodal inputs where the artificial learner, the robot, is exposed to the full bandwidth of modalities form- ing natural human-human communication in shared envi- ronments. This includes natural language utterances with references that are linguistically underspecified, the omis- sions of referents in the utterance at all, the use of pronouns without antecedents, and so forth, however, combined with body gestures such as hand-arm gestures, head movements, eye gaze, posture, etc. Taken together, the multimodal input stream, the objects and actions in a given task convey the information a listener needs in order to fully understand the verbal task description. The task scenarios for the OFAI- MMTD corpus were designed with the goal of collecting data based on which the following can be studied: • How speakers refer to objects, navigation paths etc. in a particular task setting. • What the interplay of gesture, eye gaze, and language is in a particular task demonstrated by a human tutor. • How large the inter/intra-speaker variation is when re- ferring to objects. • How often linguistic expressions are omitted such as verbs, pronouns, nouns. • What the (multimodal) cues are to prime a lis- tener/learner to pay attention to objects, paths and ac- tions. Several corpora comprising instructor-learner interactions have been collected, the majority of which are caregiver- child interactions. A large resource is the CHILDES data base which serves as a central repository for first language acquisition data. Moreover, Bj¨orkenstam and Wir´en (2013) , as well as Yu et al. (2008) collected and annotated mul- timodal caregiver-child interactions. By contrast, we are interested in the variation of the communication signals be- tween, but also within task descriptions. Thus, we need different people explain the same task, in order to better understand how humans naturally structure and present in- formation. In line with …

Content truncated to 500 words