Error-tolerant speech-to-speech translation

Error-Tolerant Speech-to-Speech Translation Rohit Kumar rkumar@bbn.com Sanjika Hewavitharana shewavit@bbn.com Nina Zinovieva nzinovie@bbn.com Matthew E. Roy mroy@bbn.com Edward Pattison-Gordon epgordon@bbn.com Raytheon BBN Technologies, 10 Moulton Street, Cambridge, MA, USA, 02138 Abstract Recent efforts to improve two-way speech-to-speech translation (S2S) systems have fo- cused on developing error detection and interactive error recovery capabilities. This article describes our current work on developing an eyes-free English-Iraqi Arabic S2S system that detects ASR errors and attempts to resolve them by eliciting user feedback. Here, we report improvements in performance across multiple system components (ASR, MT and error de- tection). We also present a controlled evaluation of the S2S system that quantifies the effect of error recovery on user effort and conversational goal achievement. 1. Introduction Over the past decade, considerable progress has been made in developing usable, two-way speech-to-speech (S2S) translation systems that enable real time cross-lingual spoken com- munication [1][2]. Conventionally, S2S systems comprise a pipeline of three speech and lan- guage technology components: automatic speech recognition (ASR), machine translation (MT) and text-to-speech synthesis (TTS). While each of these components technologies have continued to improve in performance, each is data-driven and its performance will degrade when faced with novel vocabulary. For example, large-vocabulary ASR systems are incapable of recognizing out-of-vocabulary (OOV) words, MT systems cannot translate unseen source words and TTS often mispro- nounces novel words (often high-value concepts like names and technical terminology). A combination of these deficiencies can render S2S systems unusable, especially on conversa- tional topics not well represented in the training data. Different approaches to addressing component failures in the context of S2S systems have been explored. There have also been attempts at joint optimization of ASR and MT, as well as MT and TTS [23][24][25]. Most recently, S2S system that actively detect and recover from failures have been built and evaluated [16][26]. In this work, we report our recent efforts to extend and enhance the usability of S2S translation systems through active error detection and recovery. To address system robustness, we have operationalized recent advances in ASR and sta- tistical MT (SMT) into our real-time S2S system. These advances have dramatically improved ASR and SMT performance, leading to a more robust S2S pipeline, and yet they are fast enough for real-time use. To address the problem of novel vocabulary, we have developed interfaces that enable non-technical users to rapidly enrich our S2S system with new words and phrases that are relevant to emerging use cases. Finally, we detect potential ASR errors before they are amplified by being sent through the SMT and TTS components. A simple in- Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 229 teraction strategy then allows users to correct the translation in the event of an error. Section 3 evaluates an end-to-end S2S system augmented with this capability. 2. BBN’s Speech-to-Speech Translation System 2.1. Architecture Figure 1. A unidirectional S2S system pipeline Figure 2. BBN’s S2S System with SuperMic Two-way S2S systems comprise a pair …

Content truncated to 500 words