End-to-End Speech Recognition and Disfluency Removal

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2051–2061 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2051 End-to-End Speech Recognition and Disfluency Removal Paria Jamshid Lou1 and Mark Johnson2,1 1Department of Computing, Macquarie University 2Oracle Digital Assistant, Oracle Corporation 1paria.jamshid-lou@hdr.mq.edu.au 2mark.mj.johnson@oracle.com Abstract Disfluency detection is usually an intermediate step between an automatic speech recognition (ASR) system and a downstream task. By con- trast, this paper aims to investigate the task of end-to-end speech recognition and disfluency removal. We specifically explore whether it is possible to train an ASR model to directly map disfluent speech into fluent transcripts, without relying on a separate disfluency detec- tion model. We show that end-to-end models do learn to directly generate fluent transcripts; however, their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a specialized disfluency detection model. We also propose two new metrics for evaluating integrated ASR and dis- fluency removal models. The findings of this paper can serve as a benchmark for further re- search on the task of end-to-end speech recog- nition and disfluency removal in the future. 1 Introduction Disfluency is a characteristic of spontaneous speech which is not present in written texts. Dis- fluencies include filled pauses (e.g. um and uh), repetitions (e.g. the the), corrections (e.g. Show me the flights . . . the early flights), parenthetical asides (e.g. you know), interjections (e.g. well and like), restarts (e.g. There’s a . . . Let’s go) and par- tial words (e.g. wou- and oper-) which frequently occur in spontaneous speech1 and reduce the read- ability of speech transcripts (Liu et al., 2006). They also pose a major challenge to downstream tasks relying on the output of speech recognition sys- tems, such as parsing and machine translation mod- els (Johnson and Charniak, 2004; Wang et al., 2010; Honnibal and Johnson, 2014). Since these models are usually trained on fluent clean corpora, the mis- match between the training data and the actual use 1Shriberg (1994) observed disfluencies once in every 20 words. case decreases their performance. To tackle this challenge of spontaneous speech, specialized disflu- ency detection models are developed and applied as a post-processing step to remove disfluencies from the output of speech recognition systems (Zayats et al., 2016; Wang et al., 2018; Dong et al., 2019). One type of disfluency which is especially prob- lematic for disfluency detection models is speech repair. Shriberg (1994) defines three distinct parts of a speech repair, referred to as reparandum, inter- regnum and repair. As illustrated in the example below, the reparandum to Boston is the part of the utterance that is replaced and is usually followed by an interruption point in the speech signal, the inter- regnum uh I mean is an optional part of a disfluent structure (that consists of a filled pause uh and a discourse marker I mean) and the repair to Den- ver replaces the reparandum. The fluent version is obtained by deleting reparandum and interregnum words. …

Content truncated to 500 words