Creating a Data Set of Abstractive Summaries of Turn-labeled Spoken Human-Computer Conversations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2236–2244 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2236 Creating a Data Set of Abstractive Summaries of Turn-labeled Spoken Human-Computer Conversations Virginia Meijer and Iris Hendrickx Centre for Language Studies, Centre for Language and Speech Technology Radboud University, Erasmusplein 1, 6500 HD, Nijmegen, The Netherlands virginiapatricia96@gmail.com, iris.hendrickx@ru.nl Abstract Digital recorded written and spoken dialogues are becoming increasingly available as an effect of the technological advances such as online messenger services and the use of chatbots. Summaries are a natural way of presenting the important information gathered from dialogues. We present a unique data set that consists of Dutch spoken human-computer conversations, an annotation layer of turn labels, and conversational abstractive summaries of user answers. The data set is publicly available for research purposes. We used this data set to train and evaluate two transformer abstractive summarization models for Dutch dialogues. We briefly discuss the performance and errors of these summarization models and conclude that these models are not yet sufficient to be used in practice. Keywords: automatic summarization, human-computer interaction, corpus creation 1. Introduction The aim of summarizing a piece of text is to condense the important information into a shorter and fluently written text. Humans often reuse text from the original document. Through trimming, rephrasing, and reorder- ing they compress the text into a summary containing the most important information (Jing and McKeown, 1999). In abstractive summarization, the condensed summary contains text that does not stem from the orig- inal document while extractive summaries consist of pieces of the original text. Gliwa et al. (2019) note that for certain text genres such as news articles and scien- tific publications a large number of high-quality data sets exist that include manually written summaries. However, for the emerging field of dialogue summa- rization high-quality data is less available. At the same time the authors also noted that the availability of raw digitally recorded dialogue data is increasing due to the growing popularity of messenger services and chatbots. In this paper we focus on summarizing spoken human- computer dialogues. Ultimately we would like to cre- ate an automatic summarization system that can be in- tegrated into a conversational agent. We are interested in building spoken conversational agents who talk with users about their daily life and well-being. The agent aims to create a personal profile that can provide in- sights in someone’s health and happiness (van Water- schoot et al., 2020). We believe that summaries are a natural way of inform- ing the user about what information was stored by the agent during the conversation and to provide a mecha- nism for users to signal errors in the stored information. At the end of the conversation the agent summarizes the answers of the user to the user for both transparency of the data collection process and for user feedback (Hen- drickx et al., 2021). We created a data set of summaries as a …

Content truncated to 500 words