SWING: Balancing Coverage and Faithfulness for Dialogue Summarization

Findings of the Association for Computational Linguistics: EACL 2023, pages 512–525 May 2-6, 2023 ©2023 Association for Computational Linguistics SWING : Balancing Coverage and Faithfulness for Dialogue Summarization Kung-Hsiang Huang♠∗ Siffi Singh♢ Xiaofei Ma♢ Wei Xiao♢ Feng Nan♢ Nick Dingwall♢ William Yang Wang♢ Kathleen McKeown♢ ♠University of Illinois Urbana-Champaign ♢AWS AI Labs khhuang3@illinois.edu {siffis, xiaofeim, weixiaow, nanfen, nickding, wyw, mckeownk}@amazon.com Abstract Missing information is a common issue of dia- logue summarization where some information in the reference summaries is not covered in the generated summaries. To address this issue, we propose to utilize natural language inference (NLI) models to improve coverage while avoid- ing introducing factual inconsistencies. Specifi- cally, we use NLI to compute fine-grained train- ing signals to encourage the model to generate content in the reference summaries that have not been covered, as well as to distinguish be- tween factually consistent and inconsistent gen- erated sentences. Experiments on the DIALOG- SUM and SAMSUM datasets confirm the ef- fectiveness of the proposed approach in balanc- ing coverage and faithfulness, validated with automatic metrics and human evaluations. Ad- ditionally, we compute the correlation between commonly used automatic metrics with human judgments in terms of three different dimen- sions regarding coverage and factual consis- tency to provide insight into the most suitable metric for evaluating dialogue summaries.1 1 Introduction Dialogue summarization is a text generation task that aims to produce a compact summary given a piece of conversation. Conventional approaches to dialogue summarization rely on features of conver- sation data (Goo and Chen, 2018; Li et al., 2019; Oya et al., 2014). Recently, the rise of large pre- trained language models (LMs) has enabled coher- ent and fluent summaries to be generated without these features. However, low coverage and fac- tual inconsistency remain two pressing issues as studies have shown that the summaries generated from these pre-trained LMs often do not fully cover the reference (Liu and Chen, 2021; Tang et al., ∗Work done while interning at Amazon. 1We release our source code for research purposes: https://github.com/amazon-science/AWS-SWING. GENERATED SUMMARY REFERENCE SUMMARY Take max Entailment probability 0.93 0.08 0.01 0.42 0.02 0.92 0.93 0.42 0.92 1. Charlee is in class at the university that she attends. 2. She is preparing a performance in Portuguese. 3. The writer is Mrożek. 2. He and other students are preparing a play by Mrożek translated into Portuguese. 1. Charlee is attending Portuguese theater as a subject at university. NLI Figure 1: An illustration of how NLI can help determine whether a reference sentence is covered by the generated summary. We compute the entailment probability from each reference sentence (i.e. premise) to each generated sentence (i.e. hypothesis). By taking the max value along the row dimension, the resulting vector denotes the probability that each reference sentence entails a sentence in the generated summary. In this example, the entailment probability for the second reference sentence is low, indicating that this sentence is likely not covered by the generated summary. 2022) and that the generated summaries are of- ten …

Content truncated to 500 words