We’ve had this conversation before: A Novel Approach to Measuring Dialog Similarity

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1169–1177 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1169 We’ve had this conversation before: A Novel Approach to Measuring Dialog Similarity Ofer Lavi Ella Rabinovich Segev Shlomov David Boaz Inbal Ronen Ateret Anaby-Tavor IBM Research {oferl, davidbo, inbal, atereta}@il.ibm.com {ella.rabinovich1, segev.shlomov1}@ibm.com Abstract Dialog is a core building block of human nat- ural language interactions. It contains multi- party utterances used to convey information from one party to another in a dynamic and evolving manner. The ability to compare di- alogs is beneficial in many real world use cases, such as conversation analytics for con- tact center calls and virtual agent design. We propose a novel adaptation of the edit dis- tance metric to the scenario of dialog similar- ity. Our approach takes into account various conversation aspects such as utterance seman- tics, conversation flow, and the participants. We evaluate this new approach and compare it to existing document similarity measures on two publicly available datasets. The results demonstrate that our method outperforms the other approaches in capturing dialog flow, and is better aligned with the human perception of conversation similarity. 1 Introduction Measuring semantic textual similarity lies at the heart of many natural language and text process- ing tasks, such as sentence classification, informa- tion retrieval, and question answering. Traditional text representation approaches, such as high di- mensional and sparse feature vectors, have been boosted by the introduction of efficiently learned embeddings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017), unleashing the full power of the dense semantic representa- tion of words. Subsequently, new methods were developed for contextual representation of words, sentences, paragraphs, and documents, facilitat- ing the assessment of semantic similarity between larger portions of text (Le and Mikolov, 2014; Pe- ters et al., 2018; Devlin et al., 2019). Conversations1 differ from documents that compound multiple sentences or paragraphs in a 1Although not strictly equivalent, we use the terms con- versation and dialog interchangeably hereafter. number of key ways. They are semi-structured documents constructed from a sequence of utter- ances and they present unique characteristics such as having an author for each utterance, and a con- versation flow that can be viewed as a skeleton built of dialog acts (McTear et al., 2016). Indeed, it has been shown that models adapted specifically to analyze conversations outperform those built to analyze general documents, when applied to di- alog data (Wu et al., 2020; Ohsugi et al., 2019; Henderson et al., 2020; Zhang et al., 2020). A plausible assumption, therefore, would be that di- alog similarity assessment could benefit from an approach adjusted specifically for this domain. Related work in this field is relatively sparse. Appel et al. (2018) derive two different similarity functions between conversations, one for content similarity based on TF-IDF, and the other for the conversation structure, using engineered features related to the dialog flow. The scores, computed independently for the two dimensions, are further combined to infer the …

Content truncated to 500 words