Situated Incremental Natural Language Understanding using a Multimodal, Linguistically-driven Update Model

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1803–1812, Dublin, Ireland, August 23-29 2014. Situated Incremental Natural Language Understanding using a Multimodal, Linguistically-driven Update Model Casey Kennington CITEC, Bielefeld University ckennington1 Spyros Kousidis Bielefeld University spyros.kousidis2 1@cit-ec.uni-bielefeld.de 2@uni-bielefeld.de David Schlangen Bielefeld University david.schlangen2 Abstract A common site of language use is interactive dialogue between two people situated together in shared time and space. In this paper, we present a statistical model for understanding natural human language that works incrementally (i.e., does not wait until the end of an utterance to begin processing), and is grounded by linking semantic entities with objects in a shared space. We describe our model, show how a semantic meaning representation is grounded with properties of real-world objects, and further show that it can ground with embodied, interactive cues such as pointing gestures or eye gaze. 1 Introduction Dialogue between co-located participants is possibly the most common form of language use (Clark, 1996). It is highly interactive (time is shared between two participants), interlocutors can refer to ob- jects in their visual field (space is also shared), and visual cues such as gaze or pointing gestures often play a role (shared time and space). Most computational dialogue research focuses only one of these constraints. In this paper, we present a model that processes incrementally (i.e., can potentially work interactively), can make use of the visual world by symbolically representing objects in a scene, and incorporate gaze and gestures. The model can learn from conversational data and can potentially be used in an application for a situated dialogue system, such as an autonomous robot. In the following section we will provide background and present related work. That will be followed by a description of the task and the model. In Section 4 we will show how our model performs in two experiments, the first uses speech and a visual scene, the second incorporates visual cues. 2 Background and Related Work 2.1 Background: Incremental Dialogue Processing Dialogue systems that process incrementally produce behavior that is perceived by human users to be more natural than systems that use a turn-based approach (Aist et al., 2006; Skantze and Schlangen, 2009; Skantze and Hjalmarsson, 2010). Incremental dialogue has seen improvements in speech recognition (Baumann et al., 2009), speech synthesis (Buschmeier et al., 2012), and dialogue management (Buß et al., 2010; Selfridge et al., 2012). Futhermore, architectures for incremental dialogue systems have been proposed (Schlangen and Skantze, 2009; Schlangen and Skantze, 2011) and incremental toolkits are also available (Baumann and Schlangen, 2012). In this paper, we approach natural language understanding (NLU), which aims to map an utterance to an intention, as a component in the incremental model of dialogue processing as described in (Schlangen and Skantze, 2011; Schlangen and Skantze, 2009), where incremental systems consist of a network of processing modules. Each module has a left buffer and a right buffer, where a typical module takes input This work is licensed under a Creative Commons Attribution …

Content truncated to 500 words