Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection

Proceedings of NAACL-HLT 2019, pages 86–95 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 86 Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection Vicky Zayats and Mari Ostendorf Electrical Engineering Department University of Washington [vzayats, ostendor]@uw.edu Abstract Disfluencies in spontaneous speech are known to be associated with prosodic disruptions. However, most algorithms for disfluency de- tection use only word transcripts. Integrating prosodic cues has proved difficult because of the many sources of variability affecting the acoustic correlates. This paper introduces a new approach to extracting acoustic-prosodic cues using text-based distributional prediction of acoustic cues to derive vector z-score fea- tures (innovations). We explore both early and late fusion techniques for integrating text and prosody, showing gains over a high-accuracy text-only model. 1 Introduction Speech disfluencies are frequent events in sponta- neous speech. The rate of disfluencies varies with the speaker and context; one study observed dis- fluencies once in every 20 words, affecting up to one third of utterances (Shriberg, 1994). Disflu- encies are important to account for, both because of the challenge that the disrupted grammatical flow poses for natural language processing of spo- ken transcripts and because of the information that they provide about the speaker. Most work on disfluency detection builds on the framework that annotates a disfluency in terms of a reparandum followed by an interruption point (+), an optional interregnum ({ }), and then the repair, if any. A few simple examples are given below: [ it’s + {uh} it’s] almost... [ was it, + {I mean} , did you ] put... [I just + I] enjoy working... [By + ] it was attached to... Based on the similarity/differences between the reparandum and the repair, disfluencies are often categorized into three types: repetition (the first example), rephrase (the next example), and restart (the last example). The interruption point is associated with a dis- ruption in the realization of a prosodic phrase, which could involve cutting words off or elon- gation associated with hesitation, followed by a prosodic reset at the start of the repair. There may also be emphasis in the repair to highlight the cor- rection. Researchers have been working on automatic disfluency detection for many years (Lickley, 1994; Shriberg et al., 1997; Charniak and Johnson, 2001; Johnson and Charniak, 2004; Lease et al., 2006; Qian and Liu, 2013; Zayats et al., 2016), motivated in part by early work on parsing speech that assumed reliable detection of the interruption point (Nakatani and Hirschberg, 1994; Shriberg and Stolcke, 1997; Liu et al., 2006). The first ef- forts to integrate prosody with word cues for dis- fluency detection (Baron et al., 2002; Snover et al., 2004) found gains from using prosody, but word cues played the primary role. In subsequent work (Qian and Liu, 2013; Honnibal and Johnson, 2014; Wang et al., 2017), more effective models of word transcripts have been the main source of perfor- mance gains. The success of recent neural network systems raises the question …

Content truncated to 500 words