Grounded Language Modeling for Automatic Speech Recognition of Sports Video

Proceedings of ACL-08: HLT, pages 121–129, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Grounded Language Modeling for Automatic Speech Recognition of Sports Video Michael Fleischman Massachusetts Institute of Technology Media Laboratory mbf@mit.edu Deb Roy Massachusetts Institute of Technology Media Laboratory dkroy@media.mit.edu Abstract Grounded language models represent the rela- tionship between words and the non-linguistic context in which they are said. This paper de- scribes how they are learned from large cor- pora of unlabeled video, and are applied to the task of automatic speech recognition of sports video. Results show that grounded language models improve perplexity and word error rate over text based language models, and fur- ther, support video information retrieval better than human generated speech transcriptions. 1 Introduction Recognizing speech in broadcast video is a neces- sary precursor to many multimodal applications such as video search and summarization (Snoek and Worring, 2005;). Although performance is often reasonable in controlled environments (such as studio news rooms), automatic speech recogni- tion (ASR) systems have significant difficulty in noisier settings (such as those found in live sports broadcasts) (Wactlar et al., 1996). While many researches have examined how to compensate for such noise using acoustic techniques, few have attempted to leverage information in the visual stream to improve speech recognition performance (for an exception see Murkherjee and Roy, 2003). In many types of video, however, visual context can provide valuable clues as to what has been said. For example, in video of Major League Baseball games, the likelihood of the phrase “home run” increases dramatically when a home run has actually been hit. This paper describes a method for incorporating such visual information in an ASR system for sports video. The method is based on the use of grounded language models to repre- sent the relationship between words and the non- linguistic context to which they refer (Fleischman and Roy, 2007). Grounded language models are based on re- search from cognitive science on grounded models of meaning. (for a review see Roy, 2005, and Roy and Reiter, 2005). In such models, the meaning of a word is defined by its relationship to representa- tions of the language users’ environment. Thus, for a robot operating in a laboratory setting, words for colors and shapes may be grounded in the out- puts of its computer vision system (Roy & Pent- land, 2002); while for a simulated agent operating in a virtual world, words for actions and events may be mapped to representations of the agent’s plans or goals (Fleischman & Roy, 2005). This paper extends previous work on grounded models of meaning by learning a grounded lan- guage model from naturalistic data collected from broadcast video of Major League Baseball games. A large corpus of unlabeled sports videos is col- lected and paired with closed captioning transcrip- tions of the announcers’ speech. 1 This corpus is used to train the grounded language model, which like traditional language models encode the prior probability of words for an ASR system. Unlike traditional …

Content truncated to 500 words