FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3238 - 3253 July 10-15, 2022 ©2022 Association for Computational Linguistics FACTGRAPH: Evaluating Factuality in Summarization with Semantic Graph Representations Leonardo F. R. Ribeiro†∗, Mengwen Liu‡, Iryna Gurevych†, Markus Dreyer‡, Mohit Bansal‡,§ †UKP Lab, Technical University of Darmstadt ‡Amazon Alexa AI, §UNC Chapel Hill ribeiro@aiphes.tu-darmstadt.de, {mengwliu, mddreyer, mobansal}@amazon.com gurevych@ukp.informatik.tu-darmstadt.de, mbansal@cs.unc.edu Abstract Despite recent improvements in abstractive summarization, most current approaches gener- ate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improve- ments in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FACT- GRAPH, a method that decomposes the docu- ment and the summary into structured mean- ing representations (MR), which are more suit- able for factuality evaluation. MRs describe core semantic concepts and their relations, ag- gregating the main content in both document and summary in a canonical form, and reduc- ing data sparsity. FACTGRAPH encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph con- nectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FACTGRAPH outperforms previous approaches by up to 15%. Furthermore, FACT- GRAPH improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.1 1 Introduction Recent summarization approaches based on pre- trained language models (LM) have established a new level of performance (Zhang et al., 2020; Lewis et al., 2020), generating summaries that are grammatically fluent and capable of combining salient parts of the source document. However, cur- rent models suffer from a severe limitation, generat- ing summaries that are not factually consistent, that is, the content of the summary does not meet the ∗ Work done as an intern at Amazon Alexa AI. 1Our code is publicly available at https://github. com/amazon-research/fact-graph :ARG1 :ARG0 (a) (b) […] Mr Mueller was given the role of special counsel by the justice department to lead its investigation into alleged Russian interference in last year's US election […] The NYT has reported that Mr Trump has considered firing Mr Mueller […]. (c) newspaper report consider :ARG1 :ARG0 Donald Trump fire Robert Mueller :ARG1 :ARG1 :ARG0 lead :ARG2 :ARG1 Robert Mueller investigate interference Russia allege :ARG1 US President Donald Trump has said he will fire special counsel Robert Mueller, who is investigating alleged Russian interference in the US election. :ARG0 :ARG0 Donald Trump say fire :mod :ARG0 (d) :ARG1 :ARG0 Robert Mueller :ARG1 investigate :ARG0 US elect interference Russia allege :ARG1 :ARG0 :ARG1 :ARG1 :mod US elect Figure 1: Example of (a) a document, (b) a summary, and (c) the corresponding document and (d) summary graph-based meaning representations. The summary graph does not contain the "consider" node, indicating a factual error (red dashed edge). facts of …

Content truncated to 500 words