BEA-Base: A Benchmark for ASR of Spontaneous Hungarian

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1970–1977 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1970 BEA-Base: A Benchmark for ASR of Spontaneous Hungarian Péter Mihajlik1,2, András Balog1,3, Tekla Etelka Gráczi1, Anna Kohári1, Balázs Tarján2,3, Katalin Mády1 1Hungarian Research Centre for Linguistics, 2Budapest University of Technology and Economics, 3SpeechTex Inc. 11068 Budapest, Benczúr u. 33., 21111 Budapest, Műegyetem rakpart 3., 31181 Budapest, Madách Imre utca 47. Hungary mihajlik@tmit.bme.hu, {balog.andras, graczi.tekla.etelka, kohari.anna, mady}@nytud.hu, tarjanb@tmit.bme.hu Abstract Hungarian is spoken by 15 million people, still, easily accessible Automatic Speech Recognition (ASR) benchmark datasets – especially for spontaneous speech – have been practically unavailable. In this paper, we introduce BEA-Base, a subset of the BEA spoken Hungarian database comprising mostly spontaneous speech of 140 speakers. It is built specifically to assess ASR, primarily for conversational AI applications. After defining the speech recognition subsets and task, several baselines – including classic HMM-DNN hybrid and end-to-end approaches augmented by cross-language transfer learning – are developed using open-source toolkits. The best results obtained are based on multilingual self-supervised pretraining, achieving a 45% recognition error rate reduction as compared to the classical approach – without the application of an external language model or additional supervised data. The results show the feasibility of using BEA-Base for training and evaluation of Hungarian speech recognition systems. Keywords: speech database, automatic speech recognition, spontaneous speech, evaluation 1. Introduction The development of Hungarian speech databases (Roach et al. 1996; Pollák et al., 2000; Siemund at al, 2000) and speech recognition research (Elenius & Takács, 1991; Gordos, 1991, Szarvas et al., 2000) has a long history. In practice, however, hardly any ASR result published for Hungarian is comparable to any other one in an exact way. Namely, most research groups used their in-house data without sharing them with other teams. Even if a public database was applied in a study, the selection and preprocessing of train/test sets was done usually individually, so the reconstruction of experimental setups and the reproduction of results may pose significant challenge. Thus, the lack of an easily accessible Hungarian language ASR benchmark dataset leads to a situation where speech recognition results for Hungarian are typically not directly comparable hindering the competition and the development of this important field. A refreshing attempt was made recently by Mozilla launching the Common Voice (CV) project (Ardila et al. 2019) for Hungarian, but the data collection is still in an early stage, currently only 19 hours of verified training speech data is available. We look forward to the completion of the database, although only read sentences are going to be recorded. Conversational AI and similar real-life applications, however, are much better to evaluate on spontaneous speech – obviously, no user will read his/her response to a virtual (or a real) agent. Our aim was to create a benchmark dataset for Hungarian ASR training and evaluation, focusing on the speech-to- text component of conversational AI applications. Our main contributions are the following: …

Content truncated to 500 words