The Spoken Language Understanding MEDIA Benchmark Dataset in the Era of Deep Learning: data updates, training and evaluation tools

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1595–1602 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1595 The Spoken Language Understanding MEDIA Benchmark Dataset in the Era of Deep Learning: data updates, training and evaluation tools Ga¨elle Laperri`ere1, Valentin Pelloin2, Antoine Caubri`ere1, Salima Mdhaffar1, Nathalie Camelin2, Sahar Ghannay3, Bassam Jabaian1, Yannick Est`eve1 1LIA - Avignon Universit´e, France 2LIUM - Le Mans Universit´e, France 3Universit´e Paris-Saclay, CNRS, LISN, 91405 Orsay, France 1{firstname.lastname}@univ-avignon.fr, 2{firstname.lastname}@univ-lemans.fr, 3{firstname.lastname}@limsi.fr Abstract With the emergence of neural end-to-end approaches for spoken language understanding (SLU), a growing number of studies have been presented during these last three years on this topic. The major part of these works addresses the spoken language understanding domain through a simple task like speech intent detection. In this context, new benchmark datasets have also been produced and shared with the community related to this task. In this paper, we focus on the French MEDIA SLU dataset, distributed since 2005 and used as a benchmark dataset for a large number of research works. This dataset has been shown as being the most challenging one among those accessible to the research community. Distributed by ELRA, this corpus is free for academic research since 2020. Unfortunately, the MEDIA dataset is not really used beyond the French research community. To facilitate its use, a complete recipe, including data preparation, training and evaluation scripts, has been built and integrated to SpeechBrain, an already popular open-source and all-in-one conversational AI toolkit based on PyTorch. This recipe is presented in this paper. In addition, based on the feedback of some researchers who have worked on this dataset for several years, some corrections have been brought to the initial manual annotation: the new version of the data will also be integrated into the ELRA catalogue, as the original one. More, a significant amount of data collected during the construction of the MEDIA corpus in the 2000s was never used until now: we present the first results reached on this subset — also included in the MEDIA SpeechBrain recipe — , that will be used for now as the MEDIA test2. Last, we discuss evaluation issues. Keywords: Spoken Language Understanding, Benchmark dataset, Deep learning tools 1. Introduction Spoken Language Understanding (SLU) refers to natural language processing tasks related to semantic extraction from the speech signal. It is “a field in the intersection of speech processing, natural language processing by leverag- ing technologies from machine learning and artificial in- telligence” (Tur and De Mori, 2011). For instance, in a human-machine spoken interaction system, such a task aims to convert a user input into a semantic representation according to the user’s intention and the target application domain. While treating an SLU task, three questions need to be answered: 1) how is represented the semantic of the domain? 2) Which method is the most adapted to automati- cally extract the semantic from speech and to project it into the targeted semantic representation? 3) How to …

Content truncated to 500 words