Towards Using EEG to Improve ASR Accuracy

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 382–385, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Towards Using EEG to Improve ASR Accuracy Yun-Nung Chen, Kai-Min Chang, and Jack Mostow Project LISTEN (http://www.cs.cmu.edu/∼listen) School of Computer Science, Carnegie Mellon University 5000 Forbes Avenue, Pittsburgh, PA 15213-3891, USA {yvchen,kkchang,mostow}@cs.cmu.edu Abstract We report on a pilot experiment to improve the per- formance of an automatic speech recognizer (ASR) by using a single-channel EEG signal to classify the speaker’s mental state as reading easy or hard text. We use a previously published method (Mostow et al., 2011) to train the EEG classifier. We use its prob- abilistic output to control weighted interpolation of separate language models for easy and difficult read- ing. The EEG-adapted ASR achieves higher accu- racy than two baselines. We analyze how its perfor- mance depends on EEG classification accuracy. This pilot result is a step towards improving ASR more generally by using EEG to distinguish mental states. 1 Introduction Humans use speech to communicate what’s on their mind. However, until now, automatic speech recogniz- ers (ASR) and dialogue systems have had no direct way to take into account what is going on in a speaker’s mind. Some work has attempted to infer cognitive states from volume and speaking rate to adapt language modeling (Ward and Vega, 2009) or from query click logs (Hakkani-T¨ur et al., 2011) to detect domains. new way to address this limitation is to infer mental statesA from electroencephalogram (EEG) signals. EEG is a voltage signal that can be measured on the surface of the scalp, arising from large areas of coordi- nated neural activity. This neural activity varies as a func- tion of development, mental state, and cognitive activity, and EEG can measurably detect such variation. Recently, a few companies have scaled back medical grade EEG technology to create portable EEG headsets that are commercially available and simple to use. The NeuroSky MindSetTM (2009), for example, is an audio headset equipped with a single-channel EEG sensor. It measures the voltage between an electrode that rests on the forehead and electrodes in contact with the ear. Un- like the multi-channel electrode nets worn in labs, the sensor requires no gel or saline for recording, and re- quires no expertise to wear. Even with the limitations of recording from only a single sensor and working with untrained users, Furthermore, Mostow et al.(2011) used its output signal to distinguish easy from difficult reading, achieving above-chance accuracy. Here we build on that work by using the output of such classifiers to adapt lan- guage models for ASR and thereby improve recognition accuracy. The most similar work is Jou and Schultz’s (2008) use of electromyographic (EMG) signals generated by human articulatory muscles in producing speech. They showed that augmenting acoustic features with these EMG fea- tures can achieve rudimentary silent speech detection. Pasley et al. (2012) used electrocorticographic (ECoG) recordings from nonprimary auditory cortex in the human superior …

Content truncated to 500 words