Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1479–1491 August 1–6, 2021. ©2021 Association for Computational Linguistics 1479 Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization Xiachong Feng1, Xiaocheng Feng1,2∗, Libo Qin1, Bing Qin1,2, Ting Liu1,2 1Harbin Institute of Technology, China 2Peng Cheng Laboratory, China {xiachongfeng,xcfeng,lbqin,bqin,tliu}@ir.hit.edu.cn Abstract Current dialogue summarization systems usu- ally encode the text with a number of gen- eral semantic features (e.g., keywords and top- ics) to gain more powerful dialogue modeling capabilities. However, these features are ob- tained via open-domain toolkits that are dialog- agnostic or heavily relied on human annota- tions. In this paper, we show how DialoGPT (Zhang et al., 2020b), a pre-trained model for conversational response generation, can be de- veloped as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAM- Sum and AMI, and employ pre-trained and non pre-trained models as our summarizers. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the- art performance on the SAMSum dataset1. 1 Introduction Dialogue summarization aims to generate a suc- cinct summary while retaining essential informa- tion of the dialogue (Gurevych and Strube, 2004; Chen and Yang, 2020). Theoretically, Peyrard (2019) point out that a good summary is intuitively related to three aspects, including Informativeness, Redundancy and Relevance. To this end, previous works have taken the above three aspects into account by incorporating auxil- iary annotations into the dialogue. To improve informativeness, some works annotated linguisti- cally specific words (e.g., nouns and verbs), do- main terminologies and topic words in the dialogue (Riedhammer et al., 2008; Koay et al., 2020; Zhao et al., 2020). To reduce redundancy, some works ∗Corresponding author. 1Our codes are available at: https://github.com/ xcfcode/PLM_annotator used sentence similarity-based methods to anno- tate redundant utterances. (Zechner, 2002; Murray et al., 2005). To improve relevance, some works annotated topics for the dialogue (Li et al., 2019; Liu et al., 2019; Chen and Yang, 2020). How- ever, these annotations are usually obtained via open-domain toolkits, which are not suitable for dialogues, or require manual annotations, which are labor-consuming. To alleviate the above problem, we explore the pre-trained language model as an unsupervised an- notator to automatically provide annotations for the dialogue. Recently, some works have investigated the use of pre-trained language models in an unsu- pervised manner. For example, Sainz and Rigau (2021) exploited pre-trained models for assigning domain labels to WordNet synsets. The successful recipe is that a model is obtained extensive knowl- edge via pre-training on a huge volume of data. When it comes to the dialogue domain, DialoGPT (Zhang et al., 2020b) is a SOTA conversational response generation model, which is pre-trained on the massive dialogue data. Therefore, we draw support from DialoGPT and present our DialoGPT annotator, which can perform three dialogue anno- tation tasks, …

Content truncated to 500 words