Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information

Proceedings of NAACL-HLT 2018, pages 69–81 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information Trang Tran∗1, Shubham Toshniwal∗2, Mohit Bansal3, Kevin Gimpel2, Karen Livescu2, Mari Ostendorf1 1Electrical Engineering, University of Washington 2Toyota Technological Institute at Chicago 3Department of Computer Science, UNC Chapel Hill {ttmt001, ostendor}@uw.edu, mbansal@cs.unc.edu, {shtoshni, kgimpel, klivescu}@ttic.edu Abstract In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses. For automatically parsing spo- ken utterances, we introduce a model that in- tegrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features. We find that different types of acoustic-prosodic features are individually helpful, and together give statistically significant improvements in parse and disfluency detection F1 scores over a strong text-only baseline. For this study with known sentence boundaries, error anal- yses show that the main benefit of acoustic- prosodic features is in sentences with disfluen- cies, attachment decisions are most improved, and transcription errors obscure gains from prosody. 1 Introduction While parsing has become a relatively mature tech- nology for written text, parser performance on conversational speech lags behind. Speech poses challenges for parsing: transcripts may contain er- rors and lack punctuation; even perfect transcripts can be difficult to handle because of disfluencies (restarts, repetitions, and self-corrections), filled pauses (“um”, “uh”), interjections (“like”), paren- theticals (“you know”, “I mean”), and sentence fragments. Some of these phenomena can be han- dled in standard grammars, but disfluencies typi- cally require extensions of the model. Different ap- proaches have been explored in both constituency parsing (Charniak and Johnson, 2001; Johnson and Charniak, 2004) and dependency parsing (Rasooli and Tetreault, 2013; Honnibal and Johnson, 2014). ∗Equal Contribution. Despite these challenges, speech carries helpful extra information – beyond the words – associ- ated with the prosodic structure of an utterance and encoded via variation in timing and intonation. Speakers pause in locations that are correlated with syntactic structure (Grosjean et al., 1979), and lis- teners use prosodic structure in resolving syntac- tic ambiguities (Price et al., 1991). Prosodic cues also signal disfluencies by marking the interruption point (Shriberg, 1994). However, most speech pars- ing systems in practice take little advantage of these cues. Our study focuses on this last challenge, aim- ing to incorporate prosodic cues in a neural parser, handling disfluencies as constituents via a neural attention mechanism. A challenge of incorporating prosody in pars- ing is that multiple acoustic cues interact to signal prosodic structure, including pauses, lengthening, fundamental frequency modulation, and spectral shape. These cues also vary with the phonetic seg- ment, emphasis, emotion and speaker, so feature ex- traction typically involves multiple time windows and normalization techniques. The most success- ful constituent parsers have mapped these features to prosodic boundary posteriors by using labeled training data (Kahn et al., 2005; Hale et al., 2006; Dreyer and Shafran, 2007). The approach proposed here …

Content truncated to 500 words