SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

SUMMAC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization Philippe Laban Tobias Schnabel Paul N. Bennett Marti A. Hearst UC Berkeley, USA Microsoft, USA Microsoft, USA UC Berkeley, USA∗ Abstract In the summarization domain, a key require- ment for summaries is to be factually consis- tent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for in- consistency detection, finding that past work suffered from a mismatch in input granular- ity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SUMMACCONV that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SUMMAC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SUMMACConv obtains state-of-the-art results with a balanced accu- racy of 74.4%, a 5% improvement compared with prior work. 1 Introduction Recent progress in text summarization has been remarkable, with ROUGE record-setting models published every few months, and human eval- uations indicating that automatically generated summaries are matching human-written sum- maries in terms of fluency and informativeness (Zhang et al., 2020a). A major limitation of current summarization models is their inability to remain factually consis- tent with the respective input document. Summary inconsistencies are diverse—from inversions (i.e., negation) to incorrect use of an entity (i.e., subject, object swapping), or hallucinations (i.e., intro- duction of entity not in the original document). Recent studies have shown that in some scenarios, ∗Author emails: {phillab,hearst}@berkeley.edu, {Tobias. Schnabel,Paul.N.Bennett}@microsoft.com even state-of-the-art pre-trained language mod- els can generate inconsistent summaries in more than 70% of all cases (Pagnoni et al., 2021). This has led to accelerated research around summary inconsistency detection. A closely related task to inconsistency detection is textual entailment, also referred to as Natural Language Inference (NLI), in which a hypothesis sentence must be classified as either entailed by, neutral, or contradicting a premise sentence. En- abled by the crowd-sourcing of large NLI datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), modern architectures have achieved close to human performance at the task. The similarity of NLI to inconsistency detec- tion, as well as the availability of high-performing NLI models, led to early attempts at using NLI to detect consistencyerrorsinsummaries. These early at- tempts were unsuccessful, finding that re-ranking summaries according to an NLI model can lead to an increase in consistency errors (Falke et al., 2019), or that out-of-the-box NLI models obtain 52% accuracy at the binary classification task of inconsistency detection, only slightly above random guessing (Kryscinski et al., 2020). In this work, we revisit this approach, showing that NLI models can in fact successfully be used for inconsistency detection, as long as they are used at the appropriate granularity. Figure 1 shows how crucial using the correct granularity as input …

Content truncated to 500 words