Which Model Should We Use for a Real-World Conversational Dialogue System? a Cross-Language Relevance Model or a Deep Neural Net?

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 735–742 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 735 Which Model Should We Use for a Real-World Conversational Dialogue System? a Cross-Language Relevance Model or a Deep Neural Net? Seyed Hossein Alavi, Anton Leuski, David Traum USC Institute for Creative Technologies 12015 E Waterfront Dr, Los Angeles, CA 90094 {seyedhoa, leuski, traum}@ict.usc.edu Abstract We compare two models for corpus-based selection of dialogue responses: one based on cross-language relevance with a cross-language LSTM model. Each model is tested on multiple corpora, collected from two different types of dialogue source material. Results show that while the LSTM model performs adequately on a very large corpus (millions of utterances), its performance is dominated by the cross-language relevance model for a more moderate-sized corpus (ten thousands of utterances). Keywords: Deep Neural Network, Dual-Encoder model, Cross-Language Relevance model, Ubuntu dialogue corpus, end-to-end conversational dialogue 1. Introduction End-to-end neural network models of conversational dia- logue have become increasingly popular for conversational tasks (Ritter et al., 2011; Serban et al., 2015; Zhao et al., 2017). These models eschew traditional dialogue model- ing approaches that include internal hand-crafted domain models and representations of dialogue context and sepa- rate components for understanding natural language (con- verting to the internal representation language), updating dialogue state, state-based response generation, and natu- ral language generation (Traum and Larsson, 2003; Raux et al., 2005). Instead, these models learn to respond di- rectly from a corpus, either by generating new responses or selecting a response from the corpus training data, us- ing dual encoding and hidden layers to learn appropriate dialogue continuations. However, there are still a number of questions remaining about how well such models really work for real applications, and how much data is needed to achieve acceptable performance. Other machine learn- ing approaches have been shown to be useful, with much smaller data sets. In this paper, we compare two different kinds of end-to- end system, a neural network model based on (Lowe et al., 2015) and an older kind of end-to-end dialogue model, based on cross-language retrieval (Leuski et al., 2006), im- plemented in the publicly available NPCEditor (Leuski and Traum, 2011), and previously used for systems that have been displayed in museums (Traum et al., 2012; Traum et al., 2015). We compare these models on two different datasets: the Ubuntu Corpus (Lowe et al., 2015), and one derived from one of the museum system datasets (Traum et al., 2015). In the next section, we describe the data sets we use for the experiments. In section 3 we describe the NPCEditor and its selection model. We then review some prior neural network approaches to dialogue in section 4, followed by details of the model we tested in section 5. In section 6, we describe the experiments and results. Finally, we conclude in section 7 with a discussion of the results and their implications. 2. Datasets We utilized four datasets in our …

Content truncated to 500 words