An Evaluation Framework for Multimodal Interaction

An Evaluation Framework for Multimodal Interaction Nikhil Krishnaswamy and James Pustejovsky Brandeis University Waltham, MA, USA {nkrishna, jamesp}@brandeis.edu Abstract In this paper we present a framework for evaluating interactions between a human user and an embodied virtual agent that communicates using natural language, gesture, and by executing actions in a shared context created through a visual simulation interface. These inter- actions take place in real time and demonstrate collaboration between a human and a computer on object interaction and manipulation. Our framework leverages the semantics of language and gesture to assess the level of mutual understanding during the interaction and the ease with which the two agents communicate. We present initial results from trials involving construction tasks in a blocks world scenario and discuss extensions of the evaluation framework to more naturalistic and robust interactions. Keywords: multimodal, evaluation, semantics, objects, events, language, gesture, simulation, action, vision 1. Introduction As the field of computational linguistics evolves and more sophisticated natural language systems become integrated with everyday use, naive users will come to expect their in- teractions to approximate what they are familiar with when communicating with another human, multimodally. With increased interest in multimodal interaction comes a need to evaluate the performance of a multimodal system on all lev- els with which it engages the user. Such evaluation should be modality-agnostic and assess the success of communica- tion between human and computer, based on the semantics of objects, events, and actions situated within the shared context created by the human-computer interaction. We use the modeling language VoxML (Pustejovsky and Krishnaswamy, 2016) as the platform for modeling the aforementioned objects, events, and actions, and use the VoxML-based simulation implementation VoxSim to create the environment in which a multimodal interaction involv- ing natural language and gesture takes place. This allows us to exercise VoxML object and event semantics to assess conditions on the success or failure of the interaction. 2. Multimodal Interaction A wealth of prior work exists on the role of gestural in- formation in human-computer interaction. “Put-that-there” (Bolt, 1980) included deixis for disambiguation, and in- spired a community surrounding multimodal integration (Dumas et al., 2009; Kennington et al., 2013; Turk, 2014). As speech and gesture are processed partially indepen- dently (Quek et al., 2002), using both modalities comple- mentarily increases human working memory and decreases cognitive load (Dumas et al., 2009). Visual information has been shown to be particularly useful in establishing mutual understanding that enables further communication (Clark and Wilkes-Gibbs, 1986; Clark and Brennan, 1991; Dil- lenbourg and Traum, 2006; Eisenstein et al., 2008a; Eisen- stein et al., 2008b). We will hereafter refer to this type of shared understanding as “common ground,” which can be expressed in multiple modalities. Coordination between humans using non-verbal communi- cation (cf. Cassell (2000), Cassell et al. (2000)) can be adapted to the HCI domain, particularly in the context of shared visual workspaces (Fussell et al., 2000; Kraut et al., 2003; Fussell et al., 2004; Gergle et al., 2004). Allowing for shared gaze has been …

Content truncated to 500 words