An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3122–3130 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3122 An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains Ayesha Enayet and Gita Sukthankar Department of Computer Science University of Central Florida ayeshaenayet@knights.ucf.edu, gitars@eecs.ucf.edu Abstract This paper presents an analysis of how dialogue act sequences vary across different datasets in order to anticipate the potential degradation in the performance of learned models during domain adaptation. We hypothesize the following: 1) dialogue sequences from related domains will exhibit similar n-gram frequency distributions 2) this similarity can be expressed by measuring the average Hamming distance between subsequences drawn from different datasets. Our experiments confirm that when dialogue acts sequences from two datasets are dissimilar they lie further away in embedding space, making it possible to train a classifier to discriminate between them even when the datasets are corrupted with noise. We present results from eight different datasets: SwDA, AMI (DialSum), GitHub, Hate Speech, Teams, Diplomacy Betrayal, SAMsum, and Military (Army). Our datasets were collected from many types of human communication including strategic planning, informal discussion, and social media exchanges. Our methodology provides intuition on the generalizability of dialogue models trained on different datasets. Based on our analysis, it is problematic to assume that machine learning models trained on one type of discourse will generalize well to other settings, due to contextual differences. Keywords: dialogue acts, sequence analysis, generalizability, domain shift 1. Introduction Transfer learning is commonly used in natural language processing to compensate for paucity of data; a ma- chine learning model can often be trained on a sin- gle large source dataset and then fine-tuned for smaller target datasets. Unfortunately many machine learn- ing models perform poorly when exposed to domain shifts, distributional differences between source and target datasets. Studies have shown that, unlike ma- chine learning algorithms, humans are more robust to these natural distribution shifts (Miller et al., 2020). Our research tackles the problem of learning models for discourse analysis that generalize across different com- munication settings. Discourse is often represented as a sequence of dialogue acts (DAs) where each DA repre- sents the functional purpose of the utterance in the con- versation (e.g., statement, question, agreement). Dia- logue modeling systems not only analyze the content of the utterance, but also the context of neighboring di- alogue acts to track conversational state; for instance, agreement dialogue acts often follow questions. Due to differences in the linguistic features of training and test data, natural distribution shifts may occur (Kulka- rni et al., 2020). In dialogue models that rely on the context of utterances, we hypothesize that differences in DA patterns will affect model performance. This paper presents a methodology for predicting the potential degradation in the performance of learned models during domain adaptation. Our analysis shows that dialogue sequences from related domains possess similar n-gram frequency distributions. This similarity can be quantified by measuring the average Hamming distance between subsequences drawn …

Content truncated to 500 words