Digital Voicing of Silent Speech

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5521–5530, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5521 Digital Voicing of Silent Speech David Gaddy and Dan Klein University of California, Berkeley {dgaddy,klein}@berkeley.edu Abstract In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measure- ments that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected dur- ing vocalized speech, we are the first to train from EMG collected during silently articu- lated speech. We introduce a method of train- ing on silent EMG by transferring audio tar- gets from vocalized to silent signals. Our method greatly improves intelligibility of au- dio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements. 1 Introduction In this paper, we are interested in in enabling speech-like communication without requiring sound to be produced. By using muscular sen- sor measurements of speech articulator movement, we aim to capture silent speech - utterances that have been articulated without producing sound. In particular, we focus on the task which we call dig- ital voicing, or generating synthetic speech to be transmitted or played back. Digitally voicing silent speech has a wide array of potential applications. For example, it could be used to create a device analogous to a Bluetooth headset that allows people to carry on phone conver- sations without disrupting those around them. Such a device could also be useful in settings where the environment is too loud to capture audible speech or where maintaining silence is important. Alter- natively, the technology could be used by some Figure 1: Electromyography (EMG) electrodes placed on the face can detect muscle movements from speech articulators. people who are no longer able to produce audible speech, such as individuals whose larynx has been removed due to trauma or disease (Meltzner et al., 2017). In addition to these direct uses of digital voicing for silent speech, it may also be useful as a component technology for creating silent speech- to-text systems (Schultz and Wand, 2010), making silent speech accessible to our devices and digi- tal assistants by leveraging existing high-quality audio-based speech-to-text systems. To capture information about articulator move- ment, we make use of surface electromyography (EMG). Surface EMG uses electrodes placed on top of the skin to measure electrical potentials caused by nearby muscle activity. By placing electrodes around the face and neck, we are able to capture signals from muscles in the speech articulators. Fig- ure 1 shows the EMG electrodes used to capture signals, and Figure 2 shows an example of EMG signals captured. We collect EMG measurements during both vocalized speech (normal speech pro- …

Content truncated to 500 words