CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5657 - 5668 July 10-15, 2022 ©2022 Association for Computational Linguistics CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning Xiangru Tang† Arjun Nair† Borui Wang† Bingyao Wang† Jai Desai† Aaron Wade† Haoran Li¶ Asli Celikyilmaz¶ Yashar Mehdad¶ Dragomir Radev† † Yale University ¶ Meta AI {xiangru.tang, arjun.nair, borui.wang, dragomir.radev}@yale.edu {aimeeli, aslic, mehdad}@fb.com Abstract Factual inconsistencies in generated sum- maries severely limit the practical applications of abstractive dialogue summarization. Al- though significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluation. In this work, we first devised a typology of fac- tual errors to better understand the types of hal- lucinations generated by current models and conducted human evaluation on popular dialog summarization dataset. We further propose a training strategy that improves the factual con- sistency and overall quality of summaries via a novel contrastive fine-tuning, called CON- FIT. To tackle top factual errors from our an- notation, we introduce additional contrastive loss with carefully designed hard negative sam- ples and self-supervised dialogue-specific loss to capture the key information between speak- ers. We show that our model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarization. On both datasets, we achieve significant improvements over state- of-the-art baselines using both automatic met- rics, ROUGE and BARTScore, and human evaluation. 1 Introduction Text summarization is used to generate a concise and accurate summary of a long text while focusing on the sections that convey the most useful infor- mation (Gurevych and Strube, 2004). In recent years, the resurgence of dialogue summarization has attracted significant research attentions (Mc- Cowan et al., 2005; Gliwa et al., 2019; Koay et al., 2020; Zhang et al., 2021; Zhong et al., 2021; Zhu et al., 2021; Chen et al., 2021a; Li et al., 2021; Chen et al., 2021c; Fabbri et al., 2021; Chen et al., 2021d). The goal of dialogue summarization is to condense the conversational input into brief sentences ver- sion but cover salient information (McCowan et al., 2005; Yuan and Yu, 2020). Significant progress has been made recently on abstractive dialogue summa- rization with various pre-trained models. However, such pre-trained models are susceptible to generat- ing hallucinate content that is not supported by the source documents (Cao et al., 2018; Maynez et al., 2020; Kryscinski et al., 2020). To tackle the issue of factual inconsistency in dialogue summarization, recent works correctly encode the names of speak- ers (Zhu et al., 2020), explicitly incorporate coref- erence information (Liu et al., 2021b), and order the personal named entities (Liu and Chen, 2021). But it is still challenging to improve the quality of summaries generated by different models and decrease the hallucination at the same time. To better understand the types of hallucinations generated by the pre-trained models, we devised a linguistically motivated taxonomy of factual er- rors for …

Content truncated to 500 words