Quantifying Appropriateness of Summarization Data for Curriculum Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 138–159 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 138 The Gutenberg Dialogue Dataset Richard Csaky Budapest University of Technology University of Oxford richard.csaky@psych.ox.ac.uk Gábor Recski TU Wien gabor.recski@tuwien.ac.at Abstract Large datasets are essential for neural mod- eling of many NLP tasks. Current publicly available open-domain dialogue datasets offer a trade-off between quality (e.g., DailyDialog (Li et al., 2017b)) and size (e.g., Opensubtitles (Tiedemann, 2012)). We narrow this gap by building a high-quality dataset of 14.8M utter- ances in English, and smaller datasets in Ger- man, Dutch, Spanish, Portuguese, Italian, and Hungarian. We extract and process dialogues from public-domain books made available by Project Gutenberg1. We describe our dialogue extraction pipeline, analyze the effects of the various heuristics used, and present an error analysis of extracted dialogues. Finally, we conduct experiments showing that better re- sponse quality can be achieved in zero-shot and finetuning settings by training on our data than on the larger but much noisier Opensubti- tles dataset. Our open-source pipeline2 can be extended to further languages with little addi- tional effort. Researchers can also build their versions of existing datasets by adjusting vari- ous trade-off parameters. 1 Introduction Current open-domain dialogue datasets offer trade- offs between quality and size. High-quality datasets are usually too small to represent the mul- titude of topics required for a conversational agent. Large datasets often lack good turn-segmentation and are generally noisy, models trained on such datasets generate low-quality or generic output. In Section 2 we analyze publicly available dialogue corpora and the trade-offs they offer. To address the need for large, high-quality datasets we build a corpus of 14.8M utterances in English using pub- licly available books from Project Gutenberg. We 1https://www.gutenberg.org/ 2https://github.com/ricsinaruto/ gutenberg-dialog also build datasets for German, Dutch, Spanish, Portuguese, Italian, and Hungarian, with utterance counts in the 20k–200k range. We call this dataset ensemble the Gutenberg Dialogue Dataset. We wish to make it explicit that we are not aiming to create a gold dataset. Our goal is to create a dataset which offers a better size-quality trade-off than other dialogue corpora. The Gutenberg dataset is both larger than DailyDialog (Li et al., 2017b) and has better quality than Opensubtitles (Tiede- mann, 2012), and we think it benefits researchers by filling a need in the landscape of dialogue cor- pora. The Gutenberg Dialogue Dataset and the code used to build it can be accessed through this repository: https://github.com/ricsinaruto/ gutenberg-dialog. The repository also contains all trained models presented in this paper and all data and training scripts used to produce the results. We also built a web demo interface for interacting with the trained models3. In Section 3 we offer a detailed quantitative anal- ysis of our heuristics to better understand their effects on data quality. Section 4 presents our error analysis of the English dataset both at the utterance and dialogue level. Using our MIT li- censed pipeline, researchers can easily …

Content truncated to 500 words