Abstract Meaning Representation for Gesture

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 146–153 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 146 An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets Yuqiao Wen, Guoqing Luo, Lili Mou Dept. Computing Science, Alberta Machine Intelligent Institute (Amii), University of Alberta yuqiao@ualberta.ca, gluo@ualberta.ca, doublepower.mou@gmail.com Abstract Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research. Keywords: Open-domain dialogue, Data cleaning 1. Introduction Open-domain dialogue generation is the task of gener- ating natural language utterances to converse with hu- mans (Shang et al., 2015; Mou et al., 2016; Tian et al., 2017). Such systems have wide applications in the in- dustry. For example, XiaoIce1 from Microsoft has been deployed to more than 40 platforms and has gained 660 million active users since its launch in 2014 (Zhou et al., 2020). There have been several benchmark datasets for open- domain dialogue generation, and they are largely ad- vancing the field. For example, DailyDialog (Li et al., 2017) and OpenSubtitles (Lison et al., 2018) have been extensively used in recent studies (Cai et al., 2020a; Sun et al., 2021; Zhou et al., 2021; Wang et al., 2021). However, we observe a common problem of existing open-domain dialogue datasets: the training and test sets tend to overlap with each other. That is, a large number of identical or near-identical dialogues appear in both training and test sets, probably due to mistakes in data collection and preprocessing. We further observe that such overlaps cause bizarre be- haviors in training dialogue systems: 1. Common evaluation metrics are heavily inflated. A dialogue system can achieve perfect test perfor- mance on overlapping samples by memorizing the training set. However, such high performance is fake, as it reflects the dialogue system’s ability to memorize rather than its conversational skills. 2. Reported performance based on overlapping sam- ples is arbitrary. Due to overlapping, performance may continue to grow even after 1000 epochs as the dialogue system continues to memorize more training samples. Since most researchers do not train their models long enough (e.g., for 1000 epochs), their reported performance is arbitrary depending on the maximum epoch or the early 1http://www.xiaoice.com stopping strategy, making the comparison of state- of-the-art models meaningless. 3. Generated utterances are over-informative. For example, we observe that models trained on over- lapping samples can predict the speaker’s name correctly with no context. Such behaviour is not realistic for any dialogue systems, highlighting the issues with overlapping datasets. Therefore, we argue that it is crucial to revisit and clean benchmark dialogue datasets for rigorous scientific re- search. Our contributions are …

Content truncated to 500 words