CONDUCT: An Expressive Conducting Gesture Dataset for Sound Control

CONDUCT: An Expressive Conducting Gesture Dataset for Sound Control Lei Chen, Sylvie Gibet, Camille Marteau IRISA, Universit´e Bretagne Sud Vannes, France {lei.chen, sylvie.gibet}@univ-ubs.fr, cam.marteau@hotmail.fr Abstract Recent research in music-gesture relationship has paid more attention on the sound variations and its corresponding gesture expressive- ness. In this study we are interested by gestures performed by orchestral conductors, with a focus on the expressive gestures made by the non dominant hand. We make the assumption that these gestures convey some meaning shared by most of conductors, and that they implicitly correspond to sound effects which can be encoded in musical scores. Following this hypothesis, we defined a collection of gestures for musical direction. These gestures are designed to correspond to well known functional effect on sounds, and they can be modulated to vary this effect by simply modifying one of their structural component (hand movement or hand shape). This paper presents the design of the gesture and sound sets and the protocol that has led to the database construction. The relevant musical excerpts and the related expressive gestures have been first defined by one expert musician. The gestures were then recorded through motion capture by two non experts who performed them along with recorded music. This database will serve as a basis for training gesture recognition system for live sound control and modulation. Keywords: Corpus, sound-control gestures, expressive gesture, conducting 1. Introduction The use of 3D mid-air gestures for controlling sound interactively has gained attention over this past decade (R. Aigner et al., 2012) One of the main diffi- culties to design such interfaces is the lack of guidelines to help designers in creating a set of gestures that can be significant for the given application and adopted by a large number of users. The purpose of this study is to be able to control and mod- ulate a live sound with expressive gestures. In live perfor- mances indeed the use of human movements to control both the visual and sound outputs have brought an intuitive and natural dimension to the interaction, leading to unprece- dented performances, rich in body sensations and artistic creation. Among gestures controlling sound we are seeking a subset of meaningful and expressive gestures performed in the air, without any instrument. Furthermore, we expect the message conveyed by each gesture to correspond to an underlying desired function, and the quality encoded in the movement to correspond to an understandable expressive intent. The gestures should also be sufficiently different from each other, so that they may be automatically discrim- inable and recognizable. And finally, as in high-level lan- guages, the structure of the produced gestures should allow to efficiently carry expressive variations by changing only a few gesture features such as kinematics, dynamics, geome- try, or hand shape features. Our search for a gestural language to drive sound applica- tions naturally led us to consider conducting gestures per- formed by orchestral conductors. These gestures are of particular interest, since they gather most of the mentioned properties. Moreover, they are …

Content truncated to 500 words