Correcting Errors in Speech Recognition with Articulatory Dynamics

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 60–68, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Correcting errors in speech recognition with articulatory dynamics Frank Rudzicz University of Toronto, Department of Computer Science Toronto, Ontario, Canada frank@cs.toronto.edu Abstract We introduce a novel mechanism for incorporating articulatory dynamics into speech recognition with the theory of task dynamics. This system reranks sentence- level hypotheses by the likelihoods of their hypothetical articulatory realizations which are derived from relationships learned with aligned acoustic/articulatory data. Experiments compare this with two baseline systems, namely an acoustic hid- den Markov model and a dynamic Bayes network augmented with discretized rep- resentations of the vocal tract. Our sys- tem based on task dynamics reduces word- error rates significantly by 10.2% relative to the best baseline models. 1 Introduction Although modern automatic speech recognition (ASR) takes several cues from the biological per- ception of speech, it rarely models its biological production. The result is that speech is treated as a surface acoustic phenomenon with lexical or phonetic hidden dynamics but without any phys- ical constraints in between. This omission leads to some untenable assumptions. For example, speech is often treated out of convenience as a se- quence of discrete, non-overlapping packets, such as phonemes, despite the fact that some major dif- ficulties in ASR, such as co-articulation, are by definition the result of concurrent physiological phenomena (Hardcastle and Hewlett, 1999). Many acoustic ambiguities can be resolved with knowledge of the vocal tract’s configuration (O’Shaughnessy, 2000). For example, the three nasal sonorants, /m/, /n/, and /ng/, are acousti- cally similar (i.e., they have large concentrations of energy at the same frequencies) but uniquely and reliably involve bilabial closure, tongue-tip elevation, and tongue-dorsum elevation, respec- tively. Having access to the articulatory goals of the speaker would, in theory, make the identifica- tion of linguistic intent almost trivial. Although we don’t typically have access to the vocal tract during speech recognition, its configuration can be estimated reasonably well from acoustics alone within adequate models or measurements of the vocal tract (Richmond et al., 2003; Toda et al., 2008). Evidence that such inversion takes place naturally in humans during speech perception sug- gests that the discriminability of speech sounds de- pends powerfully on their production (Liberman and Mattingly, 1985; D’Ausilio et al., 2009). This paper describes the use of explicit models of physical speech production within recognition systems. Initially, we augment traditional models of ASR with probabilistic relationships between acoustics and articulation learned from appropri- ate data. This leads to the incorporation of a high- level, goal-oriented, and control-based theory of speech production within a novel ASR system. 2 Background and related work The use of theoretical (phonological) features of the vocal tract has provided some improvement over traditional acoustic ASR systems in phoneme recognition with neural networks (Kirchhoff, 1999; Roweis, 1999), but there has been very little work in ASR informed by direct measure- ments of the vocal tract. Recently, Markov et al. (2006) …

Content truncated to 500 words