Consistent Transcription and Translation of Speech

Consistent Transcription and Translation of Speech Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, Matthias Paulik Apple {sperber,hendra,cgollan,udhay,mpaulik}@apple.com Abstract The conventional paradigm in speech trans- lation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. Although high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from incon- sistencies between both outputs that degrade the user experience. We introduce a method- ology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further intro- duce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy.1 1 Introduction Speech translation (ST) is the task of translating acoustic speech signals into text in a foreign language. According to the prevalent framing of ST (e.g., Ney, 1999), given some input speech 1We release human annotations of consistency under https://github.com/apple/ml-transcript -translation-consistency-ratings. x, ST seeks an optimal translation ˆt ∈ T , while possibly marginalizing over transcripts s ∈ S: ˆt = argmax t ∈ T {P (t | x)} (1) ≈ argmax t ∈ T �� s ∈ S PMT (t | s) PASR (s | x) � . According to this formulation, ST models primarily focus on translation quality, while tran- scription receives less emphasis. In contrast, prac- tical ST user interfaces often display transcripts to the user alongside the translations. A typical example is a two-way conversational ST applica- tion that displays the transcript to the speaker for verification, and the translation to the conversa- tion partner (Hsiao et al., 2006). Therefore, there is a mismatch between this practical requirement and the prevalent framing as described above. While traditional ST models often do commit to a single automatic speech recognition (ASR) transcript that is then passed on to a machine translation (MT) component (Stentiford and Steer, 1988; Waibel et al., 1991), researchers have undertaken much effort to mitigate resulting error propagation issues by developing models that avoid making decisions on transcripts. Recent examples include direct models (Weiss et al., 2017) that bypass transcript generation, and lattice-to-sequence models (Sperber et al., 2017) that translate the ASR search space as a whole. Despite their merits, such models may not be ideal for scenarios that display both a translation and a corresponding transcript to users. In this paper, we replace Eq. 1 by a joint transcription/translation objective to reflect this requirement: ˆs,ˆt = argmax s ∈ S,t∈ T {P (s, t | …

Content truncated to 500 words