Finding Recurrent Features of Image Schema Gestures: the FIGURE corpus

Finding Recurrent Features of Image Schema Gestures: the FIGURE corpus Andy L¨ucking, Alexander Mehler, D´esir´ee Walther, Marcel Mauri, Dennis Kurf¨urst Goethe University Frankfurt Text Technology Lab luecking@em.uni-frankfurt.de, mehler@em.uni-frankfurt.de, dwalther@informatik.uni-frankfurt.de, mauri@informatik.uni-frankfurt.de, kurfuers@informatik.uni-frankfurt.de Abstract The Frankfurt Image GestURE corpus (FIGURE) is introduced. The corpus data is collected in an experimental setting where 50 na¨ıve participants spontaneously produced gestures in response to five to six terms from a total of 27 stimulus terms. The stimulus terms have been compiled mainly from image schemata from psycholinguistics, since such schemata provide a panoply of abstract contents derived from natural language use. The gestures have been annotated for kinetic features. FIGURE aims at finding (sets of) stable kinetic feature configurations associated with the stimulus terms. Given such configurations, they can be used for designing HCI gestures that go beyond pre-defined gesture vocabularies or touchpad gestures. It is found, for instance, that movement trajectories are far more informative than handshapes, speaking against purely handshape-based HCI vocabularies. Furthermore, the mean temporal duration of hand and arm movements associated vary with the stimulus terms, indicating a dynamic dimension not covered by vocabulary-based approaches. Descriptive results are presented and related to findings from gesture studies and natural language dialogue. Keywords: gestures, human-computer interaction, kinetic features 1. Introduction Gestures for Human Computer Interaction (HCI) are usu- ally confined to navigation commands or task-oriented hand and arm movements. Thus, HCI gestures either come as manipulators or as part of a specific gesture lexicon (semaphores) (Quek et al., 2002) (cf. the touchpad ges- tures that became popular with the proliferation of touch- screens). While manipulator gestures are highly effective due to providing direct feedback (Brennan, 1998), they are strongly bound to imperative, system-controlling com- mands. Semaphores can in principle be designed as an in- put for presumably any kind of operation. However, such semaphores has to be learned in advance, hence they im- pose a high memorising demand on side of the user. In applications where more abstract declarative interaction is called for (e.g., in the context of education or museums (Mehler and L¨ucking, 2012)), rich and intuitive interac- tion means are needed. In order to meet this requirement, Mehler et al. (2014) proposed a conceptual approach to HCI gestures based on the notion of image schema propa- gated in cognitive sciences (Lakoff, 1987). This approach assumes that predicational information can be enacted by an image schema-triggering gesture whose “meaning” is derived from the underlying schema, giving rise to a kind of sign language called “gestural writing” (Mehler et al., 2014). Note that this approach is conceived for touchless HCI, as accomplished by controllers like Kinect or Leap- Motion.1 Gestural writing can be an efficient means for HCI only if there is an association between certain hand and arm movements and concepts (Grandhi et al., 2011) beyond idiosyncrasies in gesture production (Bergmann and Kopp, 2010). With regard to the coverbal use of gestures in natural lan- guage dialogues, an association has been observed between the function of a kind of gesture and certain …

Content truncated to 500 words