Learning Dialogue Representations from Consecutive Utterances

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 754 - 768 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning Dialogue Representations from Consecutive Utterances Zhihan Zhou ∗ Dejiao Zhang† Wei Xiao† Nicholas Dingwall† Xiaofei Ma† Andrew O. Arnold† Bing Xiang† ∗Northwestern University †AWS AI Labs Abstract Learning high-quality dialogue representations is essential for solving a variety of dialogue- oriented tasks, especially considering that dia- logue systems often suffer from data scarcity. In this paper, we introduce Dialogue Sen- tence Embedding (DSE), a self-supervised con- trastive learning method that learns effective di- alogue representations suitable for a wide range of dialogue tasks. DSE learns from dialogues by taking consecutive utterances1 of the same dialogue as positive pairs for contrastive learn- ing. Despite its simplicity, DSE achieves sig- nificantly better representation capability than other dialogue representation and universal sen- tence representation models. We evaluate DSE on five downstream dialogue tasks that exam- ine dialogue representation at different seman- tic granularities. Experiments in few-shot and zero-shot settings show that DSE outperforms baselines by a large margin. For example, it achieves 13% average performance improve- ment over the strongest unsupervised baseline in 1-shot intent classification on 6 datasets. 2 We also provide analyses on the benefits and limitations of our model. 1 Introduction Due to the variety of domains and the high cost of data annotation, labeled data for task-oriented dia- logue systems is often scarce or even unavailable. Therefore, learning universal dialogue representa- tions that effectively capture dialogue semantics at different granularities (Hou et al., 2020; Krone et al., 2020; Yu et al., 2021) provides a good foun- dation for solving various downstream tasks (Snell et al., 2017; Vinyals et al., 2016). Contrastive learning (Chen et al., 2020; He et al., 2020) has achieved widespread success in represen- ∗Work done during an internship at AWS AI Labs. 1Throughout this paper, we use utterance to refer to all the sentences that belong to the same dialogue turn. 2The code and pre-trained models are publicly available at https://github.com/amazon-research/dse. DSE TOD-BERT Response Selection Out-of-scope Detection SimCSE Figure 1: TSNE visualization of the dialogue repre- sentations provides by TOD-BERT, SimCSE, and DSE. Left: each color indicates one intent category, while the black circles represents out-of-scope samples. Right: items with the same color stands for query-response pairs, where triangles represent queries. The black cir- cles represents randomly sampled responses. tations learning in both the image domain (Hjelm et al., 2018; Lee et al., 2020; Bachman et al., 2019) and the text domain (Gao et al., 2021; Zhang et al., 2021a,b; Wu et al., 2020a). Contrastive learning aims to reduce the distance between semantically similar (positive) pairs and increase the distance between semantically dissimilar (negative) pairs. These positive pairs can be either human-annotated or obtained through various data augmentations, while negative pairs are often collected through negative sampling in the mini-batch. In the supervised learning regime, Gao et al. (2021); Zhang et al. (2021a) demonstrate the …

Content truncated to 500 words