SumTitles: a Summarization Dataset with Low Extractiveness

Proceedings of the 28th International Conference on Computational Linguistics, pages 5718–5730 Barcelona, Spain (Online), December 8-13, 2020 5718 SumTitles: a Summarization Dataset with Low Extractiveness Valentin Malykh Huawei Noah’s Ark lab, Moscow, Russia Konstantin Chernis Huawei Noah’s Ark lab, Moscow, Russia Ekaterina Artemova Huawei Noah’s Ark lab, HSE University Moscow, Russia Irina Piontkovskaya Huawei Noah’s Ark lab, Moscow, Russia Abstract The existing dialogue summarization corpora are significantly extractive. We introduce a method- ology for dataset extractiveness evaluation and present a new low-extractive corpus of movie dialogues for abstractive text summarization along with baseline evaluation. The corpus contains 153k dialogues and consists of three parts: 1) automatically aligned subtitles, 2) automatically aligned scenes from scripts, and 3) manually aligned scenes from scripts. We also present an alignment algorithm which we use to construct the corpus.1 1 Introduction As most written communication is held in the form of dialogues, the amount of dialogue data increases over time. This poses a requirement for efficient dialogue mining tools for information extraction, search, and natural language understanding. An attractive path towards efficient information search is a compact representation, e.g., in the form of summarization. Although summarization methods could ease processing a large amount of textual data, few are applicable to dialogues. The main reason for this is that most of the datasets used for summarization models are completely different. Probably, the most well known corpus for summarization, CNN/DailyMail (Hermann et al., 2015), comprises news articles, while other cover social media posts (V¨olske et al., 2017a) and web documents (Chen et al., 2020). Deep summarization methods, both extractive (Zhong et al., 2020) and abstractive (Lewis et al., 2019), show high performance for these datasets. Dialogue data poses new challenges: first, a dialogue presents a conversation of two or more people, while news articles or social media posts are written by one person only. This means that multiple points of view can be expressed, and all of them need to be accounted for in the summary. Second, the grammar and the structure of the utterances differ drastically: more personal pronouns and colloquial expressions are used. Finally, the conventional sentence order is distorted: two consequent sentences may not be semantically related. These challenges limit the application of extractive summarization methods and push towards abstractive ones. This paper is based upon an early work in dialogue dataset construction, namely AMI (Carletta et al., 2005). AMI corpus is small in size (141 dialogues) and does not allow the deep models training. In this paper, we try to overcome this significant flaw of AMI by developing a new large-scale dataset, a thousand times larger. We gather movies subtitles and freely available synopses into a dataset we call “corpus for Summarization of movie subTitles“ (SumTitles). Section 3 describes SumTitles in details. The construction of the SumTitles dataset is a challenging task, and we present an algorithm to create alignment for scripts in Section 4. To compare SumTitles to the other summarization datasets, we introduce a new methodology, which is described in Section …

Content truncated to 500 words