IM^2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11091 - 11103 December 7-11, 2022 ©2022 Association for Computational Linguistics IM 2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation Zhihua Jiang1, Guanghui Ye1, Dongning Rao2∗, Di Wang1, Xin Miao3 1 Department of Computer Science, Jinan University, Guangzhou 510632, P. R. China 2 School of Computer, Guangdong University of Technology, Guangzhou 510006, P. R. China 3 School of Computer Science, Wuhan University, Wuhan 430072, P.R. China tjiangzhh@jnu.edu.cn, yghljf@stu2020.jnu.edu.cn, raodn@gdut.edu.cn, windi@stu2020.jnu.edu.cn, miaoxin@whu.edu.cn Abstract Evaluation metrics shine the light on the best models and thus strongly influence the research directions, such as the recently developed dia- logue metrics USR, FED, and GRADE. How- ever, most current metrics evaluate the dia- logue data as isolated and static because they only focus on a single quality or several qual- ities. To mitigate the problem, this paper pro- poses an interpretable, multi-faceted, and con- trollable framework IM 2 (Interpretable and Multi-category Integrated Metric) to combine a large number of metrics which are good at measuring different qualities. The IM 2 frame- work first divides current popular dialogue qual- ities into different categories and then applies or proposes dialogue metrics to measure the qualities within each category and finally gen- erates an overall IM 2 score. An initial ver- sion of IM 2 was submitted to the AAAI 2022 Track5.1@DSTC10 challenge1 and took the 2nd place on both of the development and test leaderboard. After the competition, we develop more metrics and improve the performance of our model. We compare IM 2 with other 13 current dialogue metrics and experimental re- sults show that IM 2 correlates more strongly with human judgments than any of them on each evaluated dataset2. 1 Introduction Because human evaluation for natural language generation (NLG) systems is both expensive and time-consuming, relevant and meaningful auto- matic evaluation metrics that strongly correlate with human judgments are crucial. However, as ∗Corresponding author: Dongning Rao. 1The full name of Track5.1@DSTC10 is Automatic Evaluation and Moderation of Open-domain Dialogue Sys- tems (subtask 1) on the AAAI DSTC-10 (Dialog System Technology Challenges 2022) challenge. The Leaderboard: https://chateval.org/dstc10. 2Our code and data are available at: https://github.com/Jnunlplab/IM2. the one-to-many natures of dialogue makes stan- dard automatic language evaluation metrics (e.g., BLEU and METEOR) ineffective for evaluating open-domain dialogue systems (Liu et al., 2016), many automatic evaluation metrics specifically de- signed for dialogue have been recently proposed (Lan et al., 2020; Sinha et al., 2020; Huang et al., 2020; Ghazarian et al., 2020; Li et al., 2021; Mehri and Eskénazi, 2020b; Zhang et al., 2020a; Pang et al., 2020; Phy et al., 2020). Although these dialogue metrics correlate with human evaluation, they focus on a single qual- ity or a few qualities, thus evaluating the dia- logue data as isolated and static, e.g., GRADE (Huang et al., 2020) evaluates the topic coher- ence of dialogue and PredictiveEngage (Ghazar- ian et al., 2020) estimates the user engagement. Therefore, multi-quality metrics are preferred, e.g., …

Content truncated to 500 words