ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4398–4419 July 9-14, 2023 ©2023 Association for Computational Linguistics ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems Sarik Ghazarian1∗ Yijia Shao2*† Rujun Han3‡ Aram Galstyan1 Nanyun Peng4 1University of Southern California / Information Sciences Institute 2Peking University 3AWS AI Labs 4Computer Science Department of University of California, Los Angeles {sarik, galstyan}@isi.edu, shaoyj@pku.edu.cn, rujunh@amazon.com, violetpeng@cs.ucla.edu Abstract Commonsense reasoning is omnipresent in hu- man communications and thus is an impor- tant feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems is still an open challenge. We take the first step by focusing on event common- sense that considers events and their relations, and is crucial in both dialogues and general commonsense reasoning. We propose AC- CENT, an event commonsense evaluation met- ric empowered by commonsense knowledge bases (CSKBs). ACCENT first extracts event- relation tuples from a dialogue, and then eval- uates the response by scoring the tuples in terms of their compatibility with the CSKB. To evaluate ACCENT, we construct the first public event commonsense evaluation dataset for open-domain dialogues. Our experiments show that ACCENT is an efficient metric for event commonsense evaluation, which achieves higher correlations with human judgments than existing baselines. 1 Introduction Open-domain dialogue systems aim to have natural and engaging conversations with users (Chen et al., 2017). The abundance of dialogue corpus (Dziri et al., 2018) and the development of neural mod- els (Radford et al., 2019; Lewis et al., 2020) en- able open-domain dialogue systems to generate grammatically correct and meaningful responses (Zhang et al., 2020d; Bao et al., 2021; Ghazar- ian et al., 2021). Despite the success, systems still struggle to consistently produce commonsense- compliant responses as humans do. As shown in Figure 1 Example A, the generated response is not compliant with commonsense since “need ∗ Equal contribution † The work was done while the author was conducting a summer internship at UCLA. ‡ The collaboration started when the author was a gradu- ate student at USC. PersonX be allergic to paint PersonX creates art PersonX likes to paint HinderedBy to learn to paint xEffect to get a paint brush xNeed CSKB PersonX’s family mourns sad PersonX loses arms from a car accident oEffect to help PersonX oWant scared oReact CSKB Human: I boated around the world with my husband when we were younger. PLATO-2: I love boating. I also like to paint, I just need an oxygen mask. Human: I actually lost my arm from a car accident so I do not have to wear uniform at work. DiSCoL: That is interesting! Do you know Apple has anything to say on that? Example A Example B Figure 1: Examples of nonsensical system responses in open-domain dialogues. an oxygen mask” is not a reasonable prerequisite for “like to paint”. Commonsense issues for di- alogue systems can also be manifested when we consider the dialogue history. For instance, in Fig- ure 1 Example B, …

Content truncated to 500 words