Impact Analysis of the Use of Speech and Language Models Pretrained by Self-Supersivion for Spoken Language Understanding

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2949–2956 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2949 Impact Analysis of the Use of Speech and Language Models Pretrained by Self-Supersivion for Spoken Language Understanding Salima Mdhaffar1, Valentin Pelloin2, Antoine Caubri`ere1, Ga¨elle Laperri`ere1, Sahar Ghannay3, Bassam Jabaian1, Nathalie Camelin2, Yannick Est`eve1 1LIA - Avignon Universit´e, France 2LIUM - Le Mans Universit´e, France 3LISN - CNRS/Universit´e Paris-Saclay, France 1{firstname.lastname}@univ-avignon.fr, 2{firstname.lastname}@univ-lemans.fr, 3{firstname.lastname}@limsi.fr Abstract Pretrained models through self-supervised learning have been recently introduced for both acoustic and language modeling. Applied to spoken language understanding tasks, these models have shown their great potential by improving the state-of-the-art performances on challenging benchmark datasets. In this paper, we present an error analysis reached by the use of such models on the French MEDIA benchmark dataset, known as being one of the most challenging benchmarks for the slot filling task among all the benchmarks accessible to the entire research community. One year ago, the state-of-art system reached a Concept Error Rate (CER) of 13.6% through the use of an end-to-end neural architecture. Some months later, a cascade approach based on the sequential use of a fine-tuned wav2vec 2.0 model and a fine-tuned BERT model reaches a CER of 11.2%. This significant improvement raises questions about the type of errors that remain difficult to treat, but also about those that have been corrected using these models pre-trained through self-supervision learning on a large amount of data. This study brings some an- swers in order to better understand the limits of such models and open new perspectives to continue improving the performance. Keywords: Spoken Language Understanding, Slot Filling, Error Analysis, Self-supervised models 1. Introduction Spoken language understanding (SLU) aims at extract- ing a semantic representation from a speech signal in human-computer interaction applications (Tur and De Mori, 2011), like named entity recognition from speech, call routing, slot filling task in a context of human-machine dialogue,... Last years, SLU task received a lot of attention by the research community and many approaches have been proposed. Traditional SLU approaches were pro- cessed through a cascade approach that is based on the use of an automatic speech recognition (ASR) system followed by a natural language understand- ing (NLU) module applied to the automatic transcrip- tion (De Mori, 2007) from the ASR sytem. For both ASR and NLU, deep neural networks have made great advances, leading to impressive improvements of qual- itative performance for final SLU tasks (Amodei et al., 2016; Collobert et al., 2011; Vaswani et al., 2017). Nowadays, state-of-the-art SLU systems are populated with end-to-end neural approaches, based on deep neural networks, that are proposed in order to di- rectly extract semantic information from speech signal (Serdyuk et al., 2018; Ghannay et al., 2018). For cas- cade systems, the intermediate transcription may con- tain recognition errors, and the NLU module has to deal with these errors. The main advantage of end-to-end approaches is to skip the use of an intermediate speech …

Content truncated to 500 words