Edit Machines for Robust Multimodal Language Processing

Edit Machines for Robust Multimodal Language Processing Srinivas Bangalore AT&T Labs-Research 180 Park Ave Florham Park, NJ 07932 srini@research.att.com Michael Johnston AT&T Labs-Research 180 Park Ave Florham Park, NJ 07932 johnston@research.att.com Abstract Multimodal grammars provide an expres- sive formalism for multimodal integra- tion and understanding. However, hand- crafted multimodal grammars can be brit- tle with respect to unexpected, erroneous, or disfluent inputs. Spoken language (speech-only) understanding systems have addressed this issue of lack of robustness of hand-crafted grammars by exploiting classification techniques to extract fillers of a frame representation. In this paper, we illustrate the limitations of such clas- sification approaches for multimodal in- tegration and understanding and present an approach based on edit machines that combine the expressiveness of multimodal grammars with the robustness of stochas- tic language models of speech recognition. We also present an approach where the edit operations are trained from data using a noisy channel model paradigm. We eval- uate and compare the performance of the hand-crafted and learned edit machines in the context of a multimodal conversational system (MATCH). 1 Introduction Over the years, there have been several mul- timodal systems that allow input and/or output to be conveyed over multiple channels such as speech, graphics, and gesture, for example, put that there (Bolt, 1980), CUBRICON (Neal and Shapiro, 1991), QuickSet (Cohen et al., 1998), SmartKom (Wahlster, 2002), Match (Johnston et al., 2002). Multimodal integration and interpre- tation for such interfaces is elegantly expressed using multimodal grammars (Johnston and Ban- galore, 2000). These grammars support com- posite multimodal inputs by aligning speech in- put (words) and gesture input (represented as se- quences of gesture symbols) while expressing the relation between the speech and gesture input and their combined semantic representation. In (Ban- galore and Johnston, 2000; Johnston and Banga- lore, 2005), we have shown that such grammars can be compiled into finite-state transducers en- abling effective processing of lattice input from speech and gesture recognition and mutual com- pensation for errors and ambiguities. However, like other approaches based on hand- crafted grammars, multimodal grammars can be brittle with respect to extra-grammatical, erro- neous and disfluent input. For speech recognition, a corpus-driven stochastic language model (SLM) with smoothing or a combination of grammar- based and � -gram model (Bangalore and John- ston, 2004; Wang et al., 2002) can be built in order to overcome the brittleness of a grammar-based language model. Although the corpus-driven lan- guage model might recognize a user’s utterance correctly, the recognized utterance may not be assigned a semantic representation by the multi- modal grammar if the utterance is not part of the grammar. There have been two main approaches to im- proving robustness of the understanding compo- nent in the spoken language understanding litera- ture. First, a parsing-based approach attempts to recover partial parses from the parse chart when the input cannot be parsed in its entirety due to noise, in order to construct a (partial) semantic representation (Dowding et al., 1993; Allen et al., 2001; Ward, 1991). Second, a classification-based …

Content truncated to 500 words