Pretraining Methods for Dialog Context Representation Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836–3845 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3836 Pretraining Methods for Dialog Context Representation Learning Shikib Mehri∗, Evgeniia Razumovskaia∗, Tiancheng Zhao and Maxine Eskenazi Language Technologies Institute, Carnegie Mellon University {amehri,erazumov,tianchez,max+}@cs.cmu.edu Abstract This paper examines various unsupervised pretraining objectives for learning dialog con- text representations. Two novel methods of pretraining dialog context encoders are pro- posed, and a total of four methods are exam- ined. Each pretraining objective is fine-tuned and evaluated on a set of downstream dialog tasks using the MultiWoz dataset and strong performance improvement is observed. Fur- ther evaluation shows that our pretraining ob- jectives result in not only better performance, but also better convergence, models that are less data hungry and have better domain generalizability. 1 Introduction Learning meaningful representations of multi-turn dialog contexts is the cornerstone of dialog sys- tems. In order to generate an appropriate response, a system must be able to aggregate information over multiple turns, such as estimating a belief state over user goals (Williams et al., 2013) and resolving anaphora co–references (Mitkov, 2014). In the past, significant effort has gone into de- veloping better neural dialog architectures to im- prove context modeling given the same in-domain training data (Dhingra et al., 2017; Zhou et al., 2016). Recent advances in pretraining on mas- sive amounts of text data have led to state-of-the- art results on a range of natural language pro- cessing (NLP) tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018) including natural language inference, question answering and text classification. These promising results suggest a new direction for improving context modeling by creating general purpose natural language repre- sentations that are useful for many different down- stream tasks. ∗* Equal contribution. Yet pretraining methods are still in their infancy. We do not yet fully understand their properties. For example, many pretraining methods are vari- ants of language modeling (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018), e.g. predicting the previous word, next word or the masked word, given the sentence context. This approach treats natural language as a simple stream of word tokens. It relies on a com- plex model to discover high-level dependencies, through the use of massive corpora and expen- sive computation. Recently the BERT model (De- vlin et al., 2018) achieved state-of-the-art per- formance on several NLP benchmarks. It intro- duces a sentence-pair level pretraining objective, i.e. predicting whether two sentences should come after one another. This is a step towards having pretraining objectives that explicitly consider and leverage discourse-level relationships. However, it is still unclear whether language modeling is the most effective method of pretrained language rep- resentation, especially for tasks that need to ex- ploit multi-turn dependencies, e.g. dialog context modeling. Thornbury and Slade (2006) underline several discourse-level features which distinguish dialog from other types of text. Dialog must be coherent across utterance and a sequence of turns …

Content truncated to 500 words