Self-supervised Semantic-driven Phoneme Discovery for Zero-resource Speech Recognition

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8027 - 8047 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Self-supervised Semantic-driven Phoneme Discovery for Zero-resource Speech Recognition Liming Wang1, Siyuan Feng2, Mark Hasegawa-Johnson1, Chang D. Yoo3 1Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign 2Multimedia Computing Group, Delft University of Technology 3Artificial Intelligence and Machine Learning Lab, KAIST {lwang114,jhasegaw}@illinois.edu, s.feng@tudelft.nl, cd_yoo@kaist.ac.kr Abstract Phonemes are defined by their relationship to words: changing a phoneme changes the word. Learning a phoneme inventory with lit- tle supervision has been a longstanding chal- lenge with important applications to under- resourced speech technology. In this paper, we bridge the gap between the linguistic and statistical definition of phonemes and propose a novel neural discrete representation learning model for self-supervised learning of phoneme inventory with raw speech and word labels. Given the availability of phoneme segmen- tation and some mild conditions, we prove that the phoneme inventory learned by our ap- proach converges to the true one with an expo- nentially low error rate. Moreover, in experi- ments on TIMIT and Mboshi benchmarks, our approach consistently learns a better phoneme- level representation and achieves a lower er- ror rate in a zero-resource phoneme recog- nition task than previous state-of-the-art self- supervised representation learning algorithms. 1 Introduction Thanks to recent developments in self-supervised speech representation learning (van den Oord et al., 2017, 2019; Chorowski et al., 2019; Baevski et al., 2020), there is new hope for the development of speech processing systems without the need for full textual transcriptions. Supervised speech pro- cessing systems for tasks such as automatic speech recognition (ASR) rely on a large amount of tex- tual transcriptions, but self-supervised systems can be applied to under-resourced languages in which such annotation is either scarce or unavailable. A key task of the self-supervised system is to learn a discrete representation. While it is possible to dis- cretize the speech solely on the basis of its acoustic properties, a more desirable discrete representa- tion would serve as a bridge from the continuous acoustic signal toward higher-level linguistic struc- tures such as syntax and semantics. Such a rep- resentation would make it possible to repurpose algorithms developed for written languages so that they could be used for unwritten languages in tasks such as speech translation and spoken language understanding. Words are the obvious choice for a discrete, semantic-driven speech representation, but a practical speech understanding system needs at least thousands of words; learning them in an un- supervised manner may be challenging. Phonemes may be a more learnable representation. According to the standard linguistic definition, phonemes are closely linked to words: Definition 1. (Linguistic definition of phonemes (Swadesh, 1934)) Phonemes are the smallest units in speech such that given a correct native word, the replacement of one or more phonemes by other phonemes (capable of occurring in the same position) results in a native word other than that intended, or a native-like nonsense word. For …

Content truncated to 500 words