Multi-Granularity Representations of Dialog

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1752–1761, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1752 Multi-Granularity Representations of Dialog Shikib Mehri and Maxine Eskenazi Language Technologies Institute, Carnegie Mellon University {amehri,max+}@cs.cmu.edu Abstract Neural models of dialog rely on generalized latent representations of language. This paper introduces a novel training procedure which explicitly learns multiple representations of language at several levels of granularity. The multi-granularity training algorithm modifies the mechanism by which negative candidate responses are sampled in order to control the granularity of learned latent representations. Strong performance gains are observed on the next utterance retrieval task using both the MultiWOZ dataset and the Ubuntu dialog cor- pus. Analysis significantly demonstrates that multiple granularities of representation are be- ing learned, and that multi-granularity training facilitates better transfer to downstream tasks. 1 Introduction Producing generalized representations of language is a well-studied problem in natural language pro- cessing (NLP) (Montague, 1973; Davidson and Harman, 2012). Neural models typically encode an input into a latent vector, which is then used by upper layers. As such, improving the quality or generality of the learned representations will typi- cally improve performance on the final task due to the increased representative power of the model. Constructing meaningful representations of di- alog is challenging. To effectively represent the dialog context, a latent dialog representation must contain the information necessary to (1) estimate a belief state over user goals (Williams et al., 2013), (2) track entity mentions (Zhao et al., 2017), (3) resolve anaphora co-references (Mitkov, 2014), (4) model the communicative purpose of an utter- ance (Core and Allen, 1997) and (5) resolve am- biguity in natural language. A large focus area of dialog research is the development of neural archi- tectures which learn effective representations of the input (Zhou et al., 2016; Wu et al., 2016; Zhou et al., 2018). With the goal of training a model for next utterance retrieval, Zhou et al. (2018) use a deep self-attention network to produce a represen- tation of each utterance within a dialog and fol- low it with an attention between utterances and 3- D convolutional layers. Recent work has explored the use of large-scale self-supervised pre-training on very large corpora (Kiros et al., 2015; Peters et al., 2018; Devlin et al., 2018; Radford et al., 2018) as a means of improv- ing natural language representations. These pre- trained models have yielded state-of-the-art results on several downstream NLP tasks (Wang et al., 2018): text classification, natural language infer- ence, and question answering. Though such meth- ods have proven useful across several downstream tasks (Wang et al., 2018), using them for dialog re- quires expensive fine-tuning of the complex mod- els (Dinan et al., 2019; Alberti et al., 2019). The need for this fine-tuning is due to the pre-training procedure. First, the domain and style of dialog corpora differ significantly from the majority of the data used during pre-training. …

Content truncated to 500 words