An Evaluation of Alternative Strategies for Implementing Dialogue Policies Using Statistical Classification and Hand-Authored Rules

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1341–1345, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP An Evaluation of Alternative Strategies for Implementing Dialogue Policies Using Statistical Classification and Rules David DeVault, Anton Leuski, and Kenji Sagae USC Institute for Creative Technologies 12015 Waterfront Drive, Playa Vista, CA 90094 {devault,leuski,sagae}@ict.usc.edu Abstract We present and evaluate a set of archi- tectures for conversational dialogue sys- tems, exploring rule-based and statistical classification approaches. In a case study, we show that while a rule-based dialogue policy is capable of high performance if perfect natural language understanding is assumed, a direct classification approach that combines the dialogue policy with NLU has practical advantages. 1 Introduction In this paper we present and evaluate a set of alter- native dialogue system architectures that could be used to implement dialogue policies for conversa- tional characters or virtual humans. The motiva- tion for this work is to improve our understanding of the development costs and performance ben- efits associated with alternative system architec- tures for virtual human dialogue systems (Traum et al., 2005; Swartout et al., 2006; Kenny et al., 2009; Jan et al., 2009; Swartout et al., 2010). We focus on the language processing steps used in a specific virtual human system described in Section 2. We analyze the relationship between Natural Language Understanding (NLU), which maps a user’s natural language input to system- specific semantic representations, and Dialogue Management (DM), which executes a dialogue policy that dictates what the virtual human will say or do in response to the user’s input. Traditionally, designing a two step NLU+DM pipeline involves defining semantic representa- tions for the dialogue domain and writing rules that constitute the dialogue policy. This modular design has the benefit of making the DM policy easy to express in explicit rules, but carries the de- velopment cost of requiring significant linguistic expertise. Additionally, as we illustrate in this pa- per, its performance can depend critically on the reliability of the NLU module. As an alternative, we contrast this design with a direct classification approach that relies only on textual examples and effectively combines the di- alogue policy with NLU. In our case study eval- uation, we find that this approach offers superior performance, owing to the high frequency of NLU errors in the two step pipeline. The research presented in this paper extends our previous work. As we summarize in Section 2, this paper relies on the same data set and evalua- tion metric as DeVault et al. (2011), which reports results for learned policies based on maximum en- tropy models. In this paper, we add a comparison to a hand-authored policy (Rules) and a new pol- icy based on relevance models (RM). These new policies are described in Section 3. We conclude with some discussion of our new findings. 2 Research Setting and Data Set We begin by summarizing our research setting, data set, and evaluation metric. We refer the reader to DeVault et al. (2011) for …

Content truncated to 500 words