Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics

Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics Gisela Vallejo1 Timothy Baldwin1,2 Lea Frermann1 1The University of Melbourne 2MBZUAI gvallejo@student.unimelb.edu.au, {tbaldwin,lea.frermann}@unimelb.edu.au Abstract In recent years, researchers have developed question-answering based approaches to auto- matically evaluate system summaries, reporting improved validity compared to word overlap- based metrics like ROUGE, in terms of corre- lation with human ratings of criteria including fluency and hallucination. In this paper, we take a closer look at one particular metric, Quest- Eval, and ask whether: (1) it can serve as a more general metric for long document simi- larity assessment; and (2) a single correlation score between metric scores and human ratings, as the currently standard approach, is sufficient for metric validation. We find that correlation scores can be misleading, and that score dis- tributions and outliers should be taken into ac- count. With these caveats in mind, QuestEval can be a promising candidate for long docu- ment similarity assessment. 1 Introduction Methods which can provide accurate estimates of document content similarity are critical to tasks such as news analysis and fact-checking (Shaar et al., 2020). Researchers have proposed a broad range of metrics to estimate document similarity (Sai et al., 2020), from n-gram overlap metrics such as BLEU (Papineni et al., 2002) and Meteor (Lavie and Agarwal, 2007) for machine translation, and ROUGE (Lin, 2004) for automatic summarisation, to embedding-based metrics such as BERTScore (Zhang et al., 2020) and MoverScore (Zhao et al., 2019). However, these metrics have been shown to rely heavily on superficial features, correlate poorly with human annotations, and perform poorly over longer document pairs (Hanna and Bojar, 2021; Balasubramanian et al., 2020; Kryscinski et al., 2019; Koto et al., 2022). A more radical recent proposal has been to use question-answering (QA) based models (Wang et al., 2020; Scialom et al., 2021), to automatically Data Avg. Len. Doc 1 Avg. Len. Doc 2 ABC News 86 86 SemEval 535 535 SummEval 63 359 Table 1: Average document length (words) in each dataset. In the case of SummEval, Doc 1 denotes a summary while Doc 2 the source text. generate question–answer pairs from a source doc- ument, and estimate similarity by the proportion of questions that can be successfully answered on the basis of the target document. While such ap- proaches were designed to evaluate automatic sum- marisation in a reference-free manner, i.e., compare a full (long) document with its (short) summary, they can in principle be applied to arbitrary doc- ument pairs. In this paper we ask whether the QuestEval method (Scialom et al., 2021) scales to varying-length document pairs, and in particu- lar, can be used to calculate the similarity between same length documents reliably. In other words, we are comparing two evaluation settings: long– short document pairs vs. documents of the same length. In Table 1, we present the different docu- ment length scenarios in terms of average length. Consistent with other work on the evaluation of similarity metrics (including the original QuestEval paper), we …

Content truncated to 500 words