RUSAVIC Corpus: Russian Audio-Visual Speech in Cars

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1555–1559 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1555 Multi-Speaker Audio-Visual Corpus RUSAVIC: Russian Audio-Visual Speech in Cars Denis Ivanko, Dmitry Ryumin, Alexandr Axyonov, Alexey Kashevnik and Alexey Karpov St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS) St. Petersburg, Russia {ivanko.d, ryumin.d, axyonov.a, alexey.kashevnik, karpov}@iias.spb.su Abstract We present a new audio-visual speech corpus (RUSAVIC) recorded in-the-wild in a vehicle environment and designed for noise-robust speech recognition. Our goal was to produce a speech corpus which is natural (recorded in real driving conditions), controlled (providing different SNR levels by windows open/closed, moving/idling vehicle, etc.), and adequate size (the amount of data is enough to train state-of-the-art NN approaches). We focus on the problem of audio-visual speech recognition: with the use of automated lip-reading to improve the performance of audio-based speech recognition in the presence of severe acoustic noise caused by road traffic. We also describe the equipment and procedures used to create RUSAVIC corpus. Data are collected in a synchronous way through several smartphones located at different angles and equipped with FullHD video camera and microphone. The corpus includes the recordings of 20 drivers with minimum of 10 recording sessions for each. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several popular audio and visual speech recognition methods and present a set of baseline recognition results. At the moment RUSAVIC is a unique audio-visual corpus for the Russian language that is recorded in-the-wild condition and we make it publicly available. Keywords: audio-visual corpus, automatic speech recognition, data collection, automated lip-reading, driver monitoring 1. Introduction In recent years, along with the rapid development of artificial intelligence technologies, the trend to multimodality has become very important and significantly boosted machine perception. Audio and visual information represent the two main perceptual modalities that we use in our daily life. Thus, in the past decades they have been widely researched and developed by both academy and industry. Speech technology has been advanced over the last 15 years, however, despite a significant success achieved in automatic speech recognition there are still a lot of challenges when the training and test data have mismatched noise conditions such as SNR or speaking styles (Shillingford et al., 2018). This becomes especially noticeable in need of a reliable speech recognizer inside of a vehicle. Unfortunately, at the moment there is no noise- robust speech recognition system to be used in real-driving conditions. And use of a hand to control navigation system/air conditioner/smartphone may distract a driver and cause road accidents. Along with this, the acoustic noise itself is not the main challenge in this domain (Lin et al., 2018). More importantly, background noise affects not only the microphone but also it causes the speaker to increase vocal effort to overcome noise levels in his ears (so-called Lombard effect). Thus, it is not enough to simply add artificial noise to the …

Content truncated to 500 words