Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5680 - 5692 July 10-15, 2022 ©2022 Association for Computational Linguistics Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries Xiangru Tang† Alexander R. Fabbri† Ziming Mao† Griffin Adams¶ Borui Wang† Haoran Li ‡ Yashar Mehdad‡ Dragomir Radev† † Yale University ¶ Columbia University ‡ Facebook AI {xiangru.tang, alexander.fabbri, ziming.mao, borui.wang, dragomir.radev}@yale.edu griffin.adams@columbia.edu {aimeeli, mehdad}@fb.com Abstract Current pre-trained models applied for sum- marization are prone to factual inconsistencies that misrepresent the source text. Evaluating the factual consistency of summaries is thus necessary to develop better models. However, the human evaluation setup for evaluating fac- tual consistency has not been standardized. To determine the factors that affect the reliabil- ity of the human evaluation, we crowdsource evaluations for factual consistency across state- of-the-art models on two news summarization datasets using the rating-based Likert Scale and ranking-based Best-Worst Scaling. Our analy- sis reveals that the ranking-based Best-Worst Scaling offers a more reliable measure of sum- mary quality across datasets and that the reli- ability of Likert ratings highly depends on the target dataset and the evaluation design. To im- prove crowdsourcing reliability, we extend the scale of the Likert rating and present a scoring algorithm for Best-Worst Scaling that we call value learning. Our crowdsourcing guidelines will be publicly available to facilitate future work on factual consistency in summarization. 1 Introduction Pre-trained language models have achieved promising results in abstractive text summariza- tion (Edunov et al., 2019; Dong et al., 2019; Song et al., 2019; Zhang et al., 2019, 2020). A serious limitation of these models, however, is their ten- dency to produce text that is factually inconsistent with the input. Thus, evaluating the factual consis- tency of the generated summaries with respect to the source is an important task (Falke et al., 2019; Cao et al., 2020; Gabriel et al., 2021; Durmus et al., 2020; Huang et al., 2021; Pagnoni et al., 2021). Recently, metrics have been proposed for evalu- ating factual consistency, including applying natu- ral language inference (Falke et al., 2019; Kryscin- ski et al., 2020) and question-answering mod- els (Eyal et al., 2019; Scialom et al., 2019; Dur- mus et al., 2020; Wang et al., 2020). However, current metrics still do not correlate highly with human judgments on factual consistency (Koto et al., 2020; Pagnoni et al., 2021). To overcome the inherent limitation of automatic metrics, re- searchers typically crowdsource human evaluations using platforms such as Amazon’s Mechanical Turk (MTurk) (Gillick and Liu, 2010; Sabou et al., 2012; Lloret et al., 2013). However, papers often differ in their preferred evaluation protocols (Louis and Nenkova, 2013; Hardy et al., 2019). These differences in the evaluation task design affect the quality of the resulting human judgments and sys- tem comparisons (Santhanam and Shaikh, 2019). Two of the primary paradigms of crowdsourced evaluations are ranking-based and rating-based. Best-Worst Scaling (Louviere and Woodworth, 1991) is a ranking-based method by which the …

Content truncated to 500 words