Fluent Translations from Disfluent Speech in End-to-End Speech Translation

Proceedings of NAACL-HLT 2019, pages 2786–2792 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2786 Fluent Translations from Disfluent Speech in End-to-End Speech Translation Elizabeth Salesky1, Matthias Sperber2, and Alex Waibel1,2 1Carnegie Mellon University, Pittsburgh PA, U.S.A. 2Karlsruhe Institute of Technology, Karlsruhe, Germany esalesky@cs.cmu.edu Abstract Spoken language translation applications for speech suffer due to conversational speech phenomena, particularly the presence of dis- fluencies. With the rise of end-to-end speech translation models, processing steps such as disfluency removal that were previously an in- termediate step between speech recognition and machine translation need to be incorpo- rated into model architectures. We use a sequence-to-sequence model to translate from noisy, disfluent speech to fluent text with dis- fluencies removed using the recently collected ‘copy-edited’ references for the Fisher Spanish- English dataset. We are able to directly gener- ate fluent translations and introduce consider- ations about how to evaluate success on this task. This work provides a baseline for a new task, the translation of conversational speech with joint removal of disfluencies. 1 Introduction & Related Work Spoken language translation (SLT) applications suffer due to conversational speech phenomena, particularly the presence of disfluencies. In con- versational speech, speakers often use disfluencies such as filler words, repetitions, false starts, and corrections which do not naturally occur in text and may not be desired in translation outputs. Disflu- ency recognition and removal has previously been performed as an intermediate step between speech recognition (ASR) and machine translation (MT), to make disfluent ASR output better-matched to typically clean machine translation training data (Cho et al., 2013, 2014; Wang et al., 2010; Honal and Schultz, 2005; Zayats et al., 2016). With the rise of end-to-end sequence-to-sequence speech translation systems (Weiss et al., 2017; Bansal et al., 2018), disfluency removal can no longer be handled as an intermediate step between ASR and MT but needs to be incorporated into the model or handled as a post-processing step. Generating fluent translations from disfluent speech may be desired for simultaneous SLT appli- cations where removing disfluencies will improve the application’s clarity and usability. To train end- to-end speech translation requires parallel speech and text translations. This introduces data con- siderations not previously relevant with chained ASR+MT models, as different datasets could be used to train ASR and MT components. Where aligned speech and translations exist, data is typi- cally clean speech�clean text, as in news or TED talks, or disfluent speech�disfluent translations, as in Fisher or meeting data, where disfluencies were faithfully included in the references for complete- ness. While some corpora with labeled disfluen- cies exist (Cho et al., 2014; Burger et al., 2002), only subsets have been translated and/or released. Salesky et al. (2018) introduced a set of fluent refer- ences1 for Fisher Spanish-English, enabling a new task: end-to-end training and evaluation against fluent references. Previous work on disfluency removal has treated it as a sequence labeling task using word or span- level labels. However, in some cases, simply re- moving disfluencies from an …

Content truncated to 500 words