CitRet: A Hybrid Model for Cited Text Span Retrieval

Proceedings of the 29th International Conference on Computational Linguistics, pages 485–495 October 12–17, 2022. 485 SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation Longxuan Ma and Ziyu Zhuang and Weinan Zhang∗and Mingda Li and Ting Liu Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology lxma,zyzhuang,wnzhang,mdli,tliu@ir.hit.edu.cn Abstract This paper introduces a novel Self-supervised Fine-grained Dialogue Evaluation framework (SelF-Eval). The core idea is to model the correlation between turn quality and the en- tire dialogue quality. We first propose a novel automatic data construction method that can au- tomatically assign fine-grained scores for arbi- trarily dialogue data. Then we train SelF-Eval with a multi-level contrastive learning schema which helps to distinguish different score levels. Experimental results on multiple benchmarks show that SelF-Eval is highly consistent with human evaluations and better than the state- of-the-art models. We give a detailed analysis of the experiments in this paper. Our code is available on GitHub. 1 Introduction Dialogue systems (DS) aim to satisfy human needs (Shum et al., 2018; Yan, 2018; Gao et al., 2019) such as information, communication, entertain- ment, etc. Appraising the quality of the DS re- sponses reflects the system’s capability and pro- vides insights into required further improvements (Finch and Choi, 2020; Deriu et al., 2021). Among the commonly used evaluation metrics, human eval- uation is of high reliability but expensive to con- duct, automatic metrics used in language genera- tion (Perplexity (Bengio et al., 2000)) or machine translation (BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), etc.) are easy to conduct but ineffec- tive to reflect the dialogue quality (Liu et al., 2016; Novikova et al., 2017). Therefore, researchers have made great efforts to find more reliable automatic evaluation metrics that are highly correlated with human evaluation (Lowe et al., 2017; Tao et al., 2018; Mehri and Eskénazi, 2020a). The current automatic dialogue evaluation met- rics leverage semantic information (Huang et al., 2020; Mehri and Eskénazi, 2020b; Ye et al., 2021) ∗*Corresponding author 负例 负例 正例 C Ki Q C Ki Q C C Ki Q R- Ki Q R- C Ki Q R+ C Ki Q R- C Ki Q R- 负例 正例 C C Ki Q R Ki Q R C Ki Q R C Ki Q R C Ki Q R 负例 正例 C C Ki-1 Q Ki+1 Q C Ki Q C Ki+2 Q 负例 正例 C C Ki Q R- Ki Q R- C Ki Q R+ C Ki Q R- C Ki Q R- 单轮(0+0+1) 多轮(1) A1 B1 A2 B1 A1 A1 B1 B2 A2 B2 单轮(1+1+0) 多轮(2) A1 B1 A2 B1 A1 A1 B1 B2 A2 B2 单轮(0+1) 多轮(1) A1 B1 A1 A1 B1 B2 A2 B2 单轮(1+0) 多轮(1) A1 B1 A1 A1 B1 B2 A2 B2 User: "Me too, where do you usually play ? " Agent: "I always play with my friends in the park on weekends." play basketball since I was a little boy ." User: "Good, I will looking for you when I go there ." User: "I …

Content truncated to 500 words