CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2306 - 2319 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation Pei Ke1, Hao Zhou2, Yankai Lin2, Peng Li3∗, Jie Zhou2, Xiaoyan Zhu1, Minlie Huang1† 1The CoAI group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2Pattern Recognition Center, WeChat AI, Tencent Inc., China 3Institute for AI Industry Research (AIR), Tsinghua University, China kepei1106@outlook.com, {tuxzhou,yankailin,withtomzhou}@tencent.com lipeng@air.tsinghua.edu.cn, {zxy-dcs,aihuang}@tsinghua.edu.cn Abstract Existing reference-free metrics have obvious limitations for evaluating controlled text gen- eration models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judg- ments, whereas supervised ones may over- fit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by for- mulating each aspect into multiple text infill- ing tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtain- ing better generalization of evaluating gener- ated texts from different models and with dif- ferent qualities1. 1 Introduction Controlled text generation aims to generate texts under some control variables, including pre- specified content prefixes and attribute labels (such as sentiments and topics). Controlled text genera- tion has been significantly advanced by large-scale pre-trained models with respect to generation qual- ity and various control variables (Keskar et al., 2019; Dathathri et al., 2020; Yang and Klein, 2021; Liu et al., 2021a; Chan et al., 2021). Despite the great success of these generation models, it becomes critical to evaluate the qual- ity of generated texts accurately. Most of the ex- isting studies adopt unsupervised and supervised metrics to measure the quality of generated texts ∗ Part of the work was done while Peng Li was working at Tencent. † Corresponding author 1The data and codes are available at https://github. com/thu-coai/CTRLEval. under different combinations of control variables (Dathathri et al., 2020; Chan et al., 2021). The eval- uation is commonly conducted in a reference-free setting because it is challenging to collect sufficient high-quality references for each input of control variables in this open-ended text generation task (Dathathri et al., 2020). However, both unsupervised and supervised met- rics have shown limitations in the evaluation of controlled text generation: 1) Unsupervised met- rics such as perplexity (Brown et al., 1992) can only provide task-agnostic evaluation regarding the overall quality of generated texts. However, controlled text generation tasks typically involve multiple evaluation aspects (Deng et al., 2021), in- cluding the quality of generated texts themselves and the relationship between generated texts and control variables. It is thus not surprising that ex- isting unsupervised metrics without multi-aspect interpretability have low correlations with …

Content truncated to 500 words