Effectiveness of French Language Models on Abstractive Dialogue Summarization Task

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3571–3581 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3571 Effectiveness of French Language Models on Abstractive Dialogue Summarization Task Yongxin Zhou, Franc¸ois Portet, Fabien Ringeval Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG 38000 Grenoble, France {yongxin.zhou, francois.portet, fabien.ringeval}@univ-grenoble-alpes.fr Abstract Pre-trained language models have established the state-of-the-art on various natural language processing tasks, including dialogue summarization, which allows the reader to quickly access key information from long conversations in meetings, interviews or phone calls. However, such dialogues are still difficult to handle with current models because the spontaneity of the language involves expressions that are rarely present in the corpora used for pre-training the language models. Moreover, the vast majority of the work accomplished in this field has been focused on English. In this work, we present a study on the summarization of spontaneous oral dialogues in French using several language specific pre-trained models: BARThez, and BelGPT-2, as well as multilingual pre-trained models: mBART, mBARThez, and mT5. Experiments were performed on the DECODA (Call Center) dialogue corpus whose task is to generate abstractive synopses from call center conversations between a caller and one or several agents depending on the situation. Results show that the BARThez models offer the best performance far above the previous state-of-the-art on DECODA. We further discuss the limits of such pre-trained models and the challenges that must be addressed for summarizing spontaneous dialogues. Keywords: dialogue summarization, pre-trained models, conversation summarization, Non-English summarization 1. Introduction The task of automatic text summarization consists of presenting textual content in a condensed version that retains the essential information. Recently, the doc- ument summarization task has seen a sharp increase in performance due to pre-trained contextualized lan- guage models (Liu and Lapata, 2019). Unlike doc- uments, conversations involve multiple speakers, are less structured, and are composed of more informal lin- guistic usage (Sacks et al., 1978). Research on dialogue summarization have considered different domains such as summarization of meetings (Buist et al., 2004; Wang and Cardie, 2011; Oya et al., 2014) and phone calls / customer service (Tamura et al., 2011; Stepanov et al., 2015; Liu et al., 2019a). The introduction of the Transformer architecture (Vaswani et al., 2017) has encouraged the development of large-scale pre-trained language models. Compared to RNN-based models, transformer-based models have deeper structure and more parameters, they use self- attention mechanism to process each token in paral- lel and represent the relation between words. Such model have pushed the performances forward in au- tomatic summarization. For instance, on the popu- lar benchmark corpus CNN/Daily Mail (Hermann et al., 2015), (Liu and Lapata, 2019) explored fine-tuning BERT (Devlin et al., 2019) to achieve state-of-the-art performance for extractive news summarization, and BART (Lewis et al., 2020) has also improved gener- ation quality on abstractive summarization. However, for dialogue summarization, the impact of pre-trained models is still not sufficiently documented. In partic- ular, it is unclear if the pre-training dataset mismatch will …

Content truncated to 500 words