The impact of language models and loss functions on repair disfluency detection

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 703–711, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics The impact of language models and loss functions on repair disfluency detection Simon Zwarts and Mark Johnson Centre for Language Technology Macquarie University {simon.zwarts|mark.johnson|}@mq.edu.au Abstract Unrehearsed spoken language often contains disfluencies. In order to correctly inter- pret a spoken utterance, any such disfluen- cies must be identified and removed or other- wise dealt with. Operating on transcripts of speech which contain disfluencies, we study the effect of language model and loss func- tion on the performance of a linear reranker that rescores the 25-best output of a noisy- channel model. We show that language mod- els trained on large amounts of non-speech data improve performance more than a lan- guage model trained on a more modest amount of speech data, and that optimising f-score rather than log loss improves disfluency detec- tion performance. Our approach uses a log-linear reranker, oper- ating on the top n analyses of a noisy chan- nel model. We use large language models, introduce new features into this reranker and examine different optimisation strategies. We obtain a disfluency detection f-scores of 0.838 which improves upon the current state-of-the- art. 1 Introduction Most spontaneous speech contains disfluencies such as partial words, filled pauses (e.g., “uh”, “um”, “huh”), explicit editing terms (e.g., “I mean”), par- enthetical asides and repairs. Of these, repairs pose particularly difficult problems for parsing and related Natural Language Processing (NLP) tasks. This paper presents a model of disfluency detec- tion based on the noisy channel framework, which specifically targets the repair disfluencies. By com- bining language models and using an appropriate loss function in a log-linear reranker we are able to achieve f-scores which are higher than previously re- ported. Often in natural language processing algorithms, more data is more important than better algorithms (Brill and Banko, 2001). It is this insight that drives the first part of the work described in this paper. This paper investigates how we can use language models trained on large corpora to increase repair detection accuracy performance. There are three main innovations in this paper. First, we investigate the use of a variety of language models trained from text or speech corpora of vari- ous genres and sizes. The largest available language models are based on written text: we investigate the effect of written text language models as opposed to language models based on speech transcripts. Sec- ond, we develop a new set of reranker features ex- plicitly designed to capture important properties of speech repairs. Many of these features are lexically grounded and provide a large performance increase. Third, we utilise a loss function, approximate ex- pected f-score, that explicitly targets the asymmetric evaluation metrics used in the disfluency detection task. We explain how to optimise this loss func- tion, and show that this leads to a marked improve- ment in disfluency detection. This is consistent with Jansche (2005) and Smith and Eisner …

Content truncated to 500 words