A Corpus of Natural Multimodal Spatial Scene Descriptions

A Corpus of Natural Multimodal Spatial Scene Descriptions Ting Han, David Schlangen Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies Bielefeld University firstname.lastname@uni-bielefeld.de Abstract We present a corpus of multimodal spatial descriptions, as commonly occurring in route giving tasks. Participants provided natural spatial scene descriptions with speech and abstract deictic/iconic hand gestures. The scenes were composed of simple geometric objects. While the language denotes object shape and visual properties (e.g., colour), the abstract deictic gestures “placed” objects in gesture space to denote spatial relations of objects. Only together with speech do these gestures receive defined meanings. Hence, the presented corpus goes beyond previous work on gestures in multimodal interfaces that either focusses on gestures with predefined meanings (multimodal commands) or provides hand motion data without accompanying speech. At the same time, the setting is more constrained than full human/human interaction, making the resulting data more amenable to computational analysis and more directly useable for learning natural computer interfaces. Our preliminary analysis results show that co-verbal deictic gestures in the corpus reflect spatial configurations of objects, and there are variations of gesture space and verbal descriptions. The provided verbal descriptions and hand motion data will enable modelling the interpretations of natural multimodal descriptions with machine learning methods, as well as other tasks such as generating natural multimodal spatial descriptions. Keywords: Multimodal spatial descriptions, natural language, co-verbal gesture, abstract deictics 1. Introduction When describing routes that are not visible in the situated environment, humans often accompany verbal descriptions with gestures to demonstrate relative spatial relations of landmarks or trajectories of the routes to follow (Emmorey et al., 2000; Alibali, 2005; Cassell et al., 2007). For ex- ample, when trying to help a person to locate a hotel not in current view, a description might be: (1) You take the tram and get off at the stop “Schu- macher street”. Now here[deictic] is the tram station, here[deictic] is a fountain, if you walk[iconic] around it, you will see the hotel[deictic] on your left. while the verbal description specifies the landmarks and actions (i.e., tram station, walk around), the deictic ges- tures encode the spatial layout of the landmarks with posi- tion information; the iconic gesture visualises the trajectory of the route. Only when combining speech and gestures to- gether, it’s possible to form a complete interpretation of the description, making it a challenging task even for human listeners (Schneider and Taylor, 1999). In this paper, we present a corpus of multimodal spatial de- scriptions where hand gestures and speech are jointly used to describe spatial scenes. The corpus includes data col- lected from two experiments, a scene description experi- ment and a spatial description experiment. In the two exper- iments, participants received different instructions to per- form the task and got different feedback signals when per- forming the task (see Section 3.1. and Section 4.1. for de- tails). The former experiment focused on eliciting intuitive multimodal descriptions, while the latter experiment aimed to elicit spatial descriptions with human-computer interac- …

Content truncated to 500 words