Bazinga! A Dataset for Multi-Party Dialogues Structuring

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3434–3441 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3434 Bazinga! A Dataset for Multi-Party Dialogues Structuring Paul Lerner1, Juliette Bergo¨end∗, Camille Guinaudeau1, Herv´e Bredin2 Benjamin Maurice∗, Sharleyne Lefevre∗, Martin Bouteiller∗ Aman Berhe∗, L´eo Galmant∗, Ruiqing Yin∗, Claude Barras3 1Universit´e Paris-Saclay, CNRS, LISN, 2IRIT, Universit´e de Toulouse, CNRS, 3Vocapia Research 1,391400, Orsay, France, 2Toulouse, France first.last@lisn.upsaclay.fr, herve.bredin@irit.fr ∗Work done while at Universit´e Paris-Saclay, CNRS, LISN Abstract We introduce a dataset built around a large collection of TV (and movie) series. Those are filled with challenging multi-party dialogues. Moreover, TV series come with a very active fan base that allows the collection of metadata and accelerates annotation. With 16 TV and movie series, Bazinga! amounts to 400+ hours of speech and 8M+ tokens, including 500K+ tokens annotated with the speaker, addressee, and entity linking information. Along with the dataset, we also provide a baseline for speaker diarization, punctuation restoration, and person entity recognition. The results demonstrate the difficulty of the tasks and of transfer learning from models trained on mono-speaker audio or written text, which is more widely available. This work is a step towards better multi-party dialogue structuring and understanding. Bazinga! is available at hf.co/bazinga. Because (a large) part of Bazinga! is only partially annotated, we also expect this dataset to foster research towards self- or weakly- supervised learning methods. Keywords: Multi-Party Dialogues, Speaker Diarization, Entity Linking 1. Introduction Multi-party dialogues, while ubiquitous in everyday life, are understudied and lack of large-scale, open- access datasets (Mahajan and Shaikh, 2021). To fill this gap, we propose Bazinga!, a dataset of 16 TV and movie series, filled with multi-party dialogues, cover- ing a wide range of genres (e.g. drama or comedy). Although dialogues in series are mostly scripted, they emulate everyday-life situations: interruptions, speech overlap, disfluencies, etc. Each of these character- istics rises a scientific challenge (Park et al., 2022; Szyma´nski et al., 2020). The discrepancy between training datasets (either mono-speaker or written text) and human interactions (on which systems are actually deployed) has been noticed by Szyma´nski et al. (2020) in the field of Automatic Speech Recognition (ASR). They note that modern ASR systems achieve 2-3% Word Error Rate (WER) on standard datasets but peak up to the 46%–73% range on dinner party conversations. We observe a similar trend for other tasks such as named entity recognition and speaker diarization. While Bazinga! includes a gold-standard subset for evaluation, the majority of the corpus is silver-standard, i.e. annotated semi-automatically. We expect this sil- ver subset to foster research towards self- or weakly- supervised learning methods. Introduced in Section 2, the dataset provides gold- standard annotations for several tasks, ranging from speech processing to natural language processing, such as automatic speech recognition, punctuation restora- tion, named entity recognition, entity linking, ad- dressee detection, and speaker identification. The silver subset includes timestamps for every word and speaker, which allows the training and evaluation of speaker di- …

Content truncated to 500 words