DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2396–2406 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2396 DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings Che Liu, Rui Wang, Jinghua Liu, Jian Sun, Fei Huang, Luo Si DAMO Academy, Alibaba Group {liuche.lc,wr224079,shaohan.ljh,jian.sun,f.huang,luo.si}@alibaba-inc.com Abstract Learning sentence embeddings from dialogues has drawn increasing attention due to its low annotation cost and high domain adaptabil- ity. Conventional approaches employ the siamese-network for this task, which obtains the sentence embeddings through modeling the context-response semantic relevance by ap- plying a feed-forward network on top of the sentence encoders. However, as the seman- tic textual similarity is commonly measured through the element-wise distance metrics (e.g. cosine and L2 distance), such architecture yields a large gap between training and evaluat- ing. In this paper, we propose DialogueCSE, a dialogue-based contrastive learning approach to tackle this issue. DialogueCSE first in- troduces a novel matching-guided embedding (MGE) mechanism, which generates a context- aware embedding for each candidate response embedding (i.e. the context-free embedding) according to the guidance of the multi-turn context-response matching matrices. Then it pairs each context-aware embedding with its corresponding context-free embedding and fi- nally minimizes the contrastive loss across all pairs. We evaluate our model on three multi-turn dialogue datasets: the Microsoft Dialogue Corpus, the Jing Dong Dialogue Corpus, and the E-commerce Dialogue Cor- pus. Evaluation results show that our ap- proach significantly outperforms the baselines across all three datasets in terms of MAP and Spearman’s correlation measures, demonstrat- ing its effectiveness. Further quantitative ex- periments show that our approach achieves better performance when leveraging more di- alogue context and remains robust when less training data is provided. 1 Introduction Sentence embeddings are used with success for a variety of NLP applications (Cer et al., 2018) and many prior methods have been proposed with different learning schemes. Kiros et al. (2015); Lo- geswaran and Lee (2018); Hill et al. (2016) train sentence encoders in a self-supervised manner with web pages and books. Conneau et al. (2017); Cer et al. (2018); Reimers and Gurevych (2019) pro- pose to learn sentence embeddings on the super- vised datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). Although the supervised-learning approaches achieve better per- formance, they suffer from high cost of annotation in building the training dataset, which makes them hard to adapt to other domains or languages. Recently, learning sentence embeddings from dialogues has begun to attract increasing atten- tion. Dialogues provide strong semantic relation- ships among conversational utterances and are usu- ally easy to collect in large amounts. Such ad- vantages make the dialogue-based self-supervised learning methods promising to achieve compet- itive or even superior performance against the supervised-learning methods, especially under the low-resource conditions. While promising, the issue of how to effectively exploit the dialogues for this task has not been suf- ficiently explored. Yang et al. (2018) propose to train an input-response prediction model on Reddit dataset (Al-Rfou et al., 2016). Since they build their …

Content truncated to 500 words