SSR7000: A Synchronized Corpus of Ultrasound Tongue Imaging for End-to-End Silent Speech Recognition

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6866–6873 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6866 SSR7000: A Synchronized Corpus of Ultrasound Tongue Imaging for End-to-End Silent Speech Recognition Naoki Kimura, Zixiong Su, Takaaki Saeki and Jun Rekimoto The University of Tokyo, Japan {kimura-naoki, zxsu}@g.ecc.u-tokyo.ac.jp, takaaki saeki@ipc.i.u-tokyo.ac.jp, rekimoto@acm.org Abstract This article presents SSR7000, a corpus of synchronized ultrasound tongue and lip images designed for end-to-end silent speech recognition (SSR). Although neural end-to-end models are successfully updating state-of-the-art technology in the field of automatic speech recognition, SSR research based on ultrasound tongue imaging has still not evolved past cascaded DNN-HMM models due to the absence of large datasets. In this study, we constructed a large dataset, namely SSR7000, to exploit the performance of end-to-end models. The SSR7000 dataset contains ultrasound tongue and lip images of 7484 utterances by a single speaker. It contains more utterances per person than any other SSR corpus based on ultrasound imaging. We also describe preprocessing techniques to address the data variances that are inevitable when collecting a large dataset and present benchmark results using an end-to-end model. The SSR7000 corpus is publicly available under the CC BY-NC 4.0 license. Keywords: Silent speech recognition, ultrasound tongue imaging, video corpus, end-to-end speech recognition model 1. Introduction A silent speech interface (SSI) (Denby et al., 2010) en- ables us to speak or use voice interfaces without utter- ing an audible sound. The essential purpose of SSI is to expand the range of applications of voice interfaces in computing. Voice interfaces (Porcheron et al., 2018; Seaborn et al., 2021) based on automatic speech recog- nition (ASR) are intuitive interfaces that most people can use without training. It is like asking someone else to do things for us. However, the intrinsic nature of vocalization presents various constraints. For exam- ple, it is difficult to use in noisy environments. Caution is also warranted when handling information that may jeopardize privacy or confidentiality. The SSI removes this limitation by enabling non-voice interactions. It also allows communication for users in hands-busy set- tings or for those with a low voice or no voice due to tracheostomy, amyotrophic lateral sclerosis (ALS), or dysarthria. The key technology for implementing SSI is silent speech recognition (SSR). Silent speech is defined as involving only articulatory movements without vocal- ization or the use of vocal cords. Traditionally, sen- sors such as surface electromyography (Maier-Hein et al., 2005; Kapur et al., 2018), electroencephalography (Porbadnigk et al., 2009), a front camera for lip read- ing (Wand et al., 2016; Assael et al., 2016b; Sun et al., 2018), and ultrasound imaging (Kimura et al., 2019; Cai et al., 2011; Ji et al., 2018a) have been used for SSR. Among these, ultrasound imaging is superior as a non-invasive and safe means of obtaining detailed im- ages of the body, as it is also used during pregnancy (Denby et al., 2010). It can capture tongue movements, which play a vital role in …

Content truncated to 500 words