Multimodal Grammar Implementation

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 582–586, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Multimodal Grammar Implementation Katya Alahverdzhieva University of Edinburgh 10 Crichton Street Edinburgh EH8 9AB K.Alahverdzhieva@sms.ed.ac.uk Dan Flickinger Stanford University Stanford, CA 94305-2150 danf@stanford.edu Alex Lascarides University of Edinburgh 10 Crichton Street Edinburgh EH8 9AB alex@inf.ed.ac.uk Abstract This paper reports on an implementation of a multimodal grammar of speech and co-speech gesture within the LKB/PET grammar engi- neering environment. The implementation ex- tends the English Resource Grammar (ERG, Flickinger (2000)) with HPSG types and rules that capture the form of the linguistic signal, the form of the gestural signal and their rel- ative timing to constrain the meaning of the multimodal action. The grammar yields a sin- gle parse tree that integrates the spoken and gestural modality thereby drawing on stan- dard semantic composition techniques to de- rive the multimodal meaning representation. Using the current machinery, the main chal- lenge for the grammar engineer is the non- linear input: the modalities can overlap tem- porally. We capture this by identical speech and gesture token edges. Further, the semantic contribution of gestures is encoded by lexical rules transforming a speech phrase into a mul- timodal entity of conjoined spoken and gestu- ral semantics. 1 Introduction Our aim is to regiment the form-meaning mapping of multimodal actions consisting of speech and co- speech gestures. The language of study is English, and the gestures of interest are depicting—the hand depicts the referent—and deictic—the hand points at the referent’s spatial coordinates. Motivation for encoding the form-meaning map- ping in the grammar stems from the fact that form effects judgments of multimodal grammaticality: e.g., in (1)1 the gesture performance along with 1The speech item where the gesture is performed is marked by underlining, and the accented item is given in uppercase. the unaccented “called” in a single prosodic phrase seems ill-formed despite the gesture depicting an as- pect of the referent—the act of calling. (1) * Your MOTHER called ... Hand lifts to the ear to imitate holding a receiver. This intuitive judgment is in line with the em- pirical findings of Giorgolo and Verstraten (2008) who observed that prosody influences the perception of temporally misaligned speech-and-gesture sig- nals as ill-formed. Further, Alahverdzhieva and Las- carides (2010) established empirically that the ges- ture performance can be predicted from the prosodic prominence in speech and that gestures not overlap- ping subject NPs cannot be semantically related with that subject NP. The fact that speech-and-gesture in- tegration is informed by the form of the linguistic signal suggests formalising the integration within the grammar. Alternatively, integrating the gestu- ral contribution by discourse update would involve pragmatic reasoning accessing information about linguistic form, disrupting the transition between syntax/semantics and pragmatics. The work is set within HPSG — a constraint-based grammar framework with the different types and rules organised in a hierarchy. The semantic infor- mation, derived in parallel with syntax, is expressed …

Content truncated to 500 words