The Vocal Joystick: A Voice-Based Human-Computer Interface for Individuals with Motor Impairments

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 995–1002, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics The Vocal Joystick: A Voice-Based Human-Computer Interface for Individuals with Motor Impairments∗ Jeff A. Bilmes†, Xiao Li†, Jonathan Malkin†, Kelley Kilanski‡, Richard Wright‡, Katrin Kirchhoff†, Amarnag Subramanya†, Susumu Harada§, James A. Landay§, Patricia Dowden¶, Howard Chizeck† †Dept. of Electrical Engineering §Dept. of Computer Science & Eng. ‡Dept. of Linguistics ¶Dept. of Speech & Hearing Science University of Washington Seattle, WA Abstract We present a novel voice-based human- computer interface designed to enable in- dividuals with motor impairments to use vocal parameters for continuous control tasks. Since discrete spoken commands are ill-suited to such tasks, our interface exploits a large set of continuous acoustic- phonetic parameters like pitch, loudness, vowel quality, etc. Their selection is opti- mized with respect to automatic recogniz- ability, communication bandwidth, learn- ability, suitability, and ease of use. Pa- rameters are extracted in real time, trans- formed via adaptation and acceleration, and converted into continuous control sig- nals. This paper describes the basic en- gine, prototype applications (in particu- lar, voice-based web browsing and a con- trolled trajectory-following task), and ini- tial user studies confirming the feasibility of this technology. 1 Introduction Many existing human-computer interfaces (e.g., mouse and keyboard, touch screens, pen tablets, etc.) are ill-suited to individuals with motor impairments. Specialized (and often expensive) human-computer interfaces that have been devel- oped specifically for this target group include sip and puff switches, head mice, eye-gaze devices, chin joysticks and tongue switches. While many indi- viduals with motor impairments have complete use ∗This material is based on work supported by the National Science Foundation under grant IIS-0326382. of their vocal system, these devices make little use of it. Sip and puff switches, for example, have low communication bandwidth, making it impossible to achieve more complex control tasks. Natural spoken language is often regarded as the obvious choice for a human-computer inter- face. However, despite significant research efforts in automatic speech recognition (ASR) (Huang et al., 2001), existing ASR systems are still not suf- ficiently robust to a wide variety of speaking condi- tions, noise, accented speakers, etc. ASR-based in- terfaces are therefore often abandoned by users after a short initial trial period. In addition, natural speech is optimal for communication between humans but sub-optimal for manipulating computers, windows- icons-mouse-pointer (WIMP) interfaces, or other electro-mechanical devices (such as a prosthetic ro- botic arm). Standard spoken language commands are useful for discrete but not for continuous op- erations. For example, in order to move a cursor from the bottom-left to the upper-right of a screen, the user might have to repeatedly utter “up” and “right” or “stop” and “go” after setting an initial tra- jectory and rate, which is quite inefficient. For these reasons, we are developing alternative and reusable voice-based assistive technology termed the “Vocal Joystick” (VJ). 2 The Vocal Joystick The VJ approach has three main characteristics: 1) Continuous control parameters: …

Content truncated to 500 words