FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10469 - 10483 December 7-11, 2022 ©2022 Association for Computational Linguistics FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows Jianqiao Zhao1∗, Yanyang Li1∗, Wanyu Du3∗, Yangfeng Ji3, Dong Yu4, Michael R. Lyu1, Liwei Wang12† 1Department of Computer Science and Engineering, The Chinese University of Hong Kong 2Shanghai AI Laboratory 3Department of Computer Science, University of Virginia 4Tencent AI Lab, Bellevue {jqzhao, yyli21, lyu, lwwang}@cse.cuhk.edu.hk Abstract Despite recent progress in open-domain dia- logue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly ex- plicitly modeled in previous methods. How- ever, defined at the utterance level in general, dialog act is of coarse granularity, as an utter- ance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large- scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue eval- uation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Ex- tensive experiments against strong baselines on three benchmark datasets demonstrate the ef- fectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation. 1 Introduction Dialogue evaluation plays a crucial role in the re- cent advancement of dialogue research. While hu- man evaluation is often considered as a universal and reliable method by the community (Smith et al., 2022), automatic dialogue evaluation metrics draw growing attention as they can assess dialogues with faster speed and lower cost (Tao et al., 2018; Huang et al., 2020; Mehri and Eskénazi, 2020). Traditional word-overlap metrics, like BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005), lose some of their effectiveness in the dialogue setting as reliable references are hard to obtain (Liu et al., 2016). Recent works tackle this problem by leveraging more sophisti- cated architectures (Zhang et al., 2021; Li et al., ∗Equal contributions. Wanyu participated in building the dataset, while doing her internship with Prof. Liwei Wang. †Corresponding author. Dialogue Section Segment Act Flow Speaker1: How are you? May I have a cup of coffee? greeting , directive Speaker2: Hmm. Certainly. What kind of coffee do you like? We have espresso and latte. backchannel- success , commissive , question , inform Table 1: A snippet of an open-domain dialogue and its segment act flow. Each segment is marked with the same color as its corresponding segment act label. 2021) and harnessing the power of large models (Mehri and Eskénazi, 2020). Although these re- cent metrics claim to show some progress towards higher correlation with humans, the gap between automatic metrics and human evaluation is still no- ticeable (Yeh et al., 2021). Automatic open-domain dialogue evaluation is still an open question, and extensive efforts have been made to improve per- formance from different …

Content truncated to 500 words