HaRiM+: Evaluating Summary Quality with Hallucination Risk

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 895–924 November 20–23, 2022. ©2022 Association for Computational Linguistics 895 HaRiM+: Evaluating Summary Quality with Hallucination Risk Seonil Son, Junsoo Park, Jeong-in Hwang, Junghwa Lee, Hyungjong Noh, Yeonsoo Lee NCSOFT NLP Center {deftson,junsoopark,jihwang,jleehhh0217,nohhj0209,yeonsoo}@ncsoft.com Abstract One of the challenges of developing a sum- marization model arises from the difficulty in measuring the factual inconsistency of the gen- erated text. In this study, we reinterpret the decoder overconfidence-regularizing objective suggested in (Miao et al., 2021) as a hallu- cination risk measurement to better estimate the quality of generated summaries. We pro- pose a reference-free metric, HaRiM+, which only requires an off-the-shelf summarization model to compute the hallucination risk based on token likelihoods. Deploying it requires no additional training of models or ad-hoc mod- ules, which usually need alignment to human judgments. For summary-quality estimation, HaRiM+ records state-of-the-art correlation to human judgment on three summary-quality an- notation sets: FRANK, QAGS, and SummEval. We hope that our work, which merits the use of summarization models, facilitates the progress of both automated evaluation and generation of summary. 1 Introduction Although recent state-of-the-art summarization models have achieved remarkable performances (Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020), appropriate metrics for measuring faithful- ness of the generated summaries are still needed. The practice of measuring performance in the sum- marization task heavily relies on the N-gram match- ing based metric, ROUGE (Lin, 2004). Reportedly, ROUGE barely satisfies more than indicating lex- ical similarity (Maynez et al., 2020) and does not consider semantic dimensions of the generation, which current research needs of. There have been numerous attempts to come up with faithfulness evaluation metrics (Novikova et al., 2017; Peyrard, 2019). Neural-based met- rics have demonstrated good performances in es- timating the factual consistency of a summary- article pair with semantic entailment (Kryscinski et al., 2020; Goyal and Durrett, 2020), question- answering framework (Wang et al., 2020; Scialom et al., 2021, 2019), and text generation (Yuan et al., 2021; Xie et al., 2021). Most of the model-as- a-metric approach generally requires fine-tuning or complicated pipelines. Consequently, evaluat- ing generated texts with recent model-as-a-metric methods has become cumbersome. With the increased demand for faithful genera- tion models, it has come to a lot of attention on reformulating training objectives for purported for this (Zhang et al., 2022; Liu et al., 2022a; Holtzman et al., 2018). We focus on the training objective suggested in (Miao et al., 2021), which directly targets hallucination problems in generating sen- tences given a source context. Miao et al. suggest that an overconfident decoder causes hallucination since the model excessively pays attention to the previously generated tokens over the source con- text which is in line with (Bowman et al., 2016). In this paper, we reinterpret the decoder overcon- fidence regularization term from (Miao et al., 2021) as hallucination risk and recompose …

Content truncated to 500 words