Real-time Incremental Speech-to-Speech Translation of Dialogs

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 437–445, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Real-time Incremental Speech-to-Speech Translation of Dialogs Srinivas Bangalore, Vivek Kumar Rangarajan Sridhar, Prakash Kolan Ladan Golipour, Aura Jimenez AT&T Labs - Research 180 Park Avenue Florham Park, NJ 07932, USA vkumar,srini,pkolan,ladan,aura@research.att.com Abstract In a conventional telephone conversation be- tween two speakers of the same language, the interaction is real-time and the speakers pro- cess the information stream incrementally. In this work, we address the problem of incre- mental speech-to-speech translation (S2S) that enables cross-lingual communication between two remote participants over a telephone. We investigate the problem in a novel real-time Session Initiation Protocol (SIP) based S2S framework. The speech translation is per- formed incrementally based on generation of partial hypotheses from speech recognition. We describe the statistical models comprising the S2S system and the SIP architecture for enabling real-time two-way cross-lingual dia- log. We present dialog experiments performed in this framework and study the tradeoff in ac- curacy versus latency in incremental speech translation. Experimental results demonstrate that high quality translations can be generated with the incremental approach with approxi- mately half the latency associated with non- incremental approach. 1 Introduction In recent years, speech-to-speech translation (S2S) technology has played an increasingly important role in narrowing the language barrier in cross- lingual interpersonal communication. The improve- ments in automatic speech recognition (ASR), statis- tical machine translation (MT), and, text-to-speech synthesis (TTS) technology has facilitated the serial binding of these individual components to achieve S2S translation of acceptable quality. Prior work on S2S translation has primarily fo- cused on providing either one-way or two-way trans- lation on a single device (Waibel et al., 2003; Zhou et al., 2003). Typically, the user interface requires the participant(s) to choose the source and target lan- guage apriori. The nature of communication, either single user talking or turn taking between two users can result in a one-way or cross-lingual dialog inter- action. In most systems, the necessity to choose the directionality of translation for each turn does take away from a natural dialog flow. Furthermore, single interface based S2S translation (embedded or cloud- based) is not suitable for cross-lingual communica- tion when participants are geographically distant, a scenario more likely in a global setting. In such a scenario, it is imperative to provide real-time and low latency communication. In a conventional telephone conversation between two speakers of the same language, the interaction is real-time and the speakers process the informa- tion stream incrementally. Similarly, cross-lingual dialog between two remote participants will greatly benefit through incremental translation. While in- cremental decoding for text translation has been addressed previously in (Furuse and Iida, 1996; Sankaran et al., 2010), we address the problem in a speech-to-speech translation setting for enabling real-time cross-lingual dialog. We address the prob- lem of incrementality in a novel session initiation protocol (SIP) based S2S translation system that en- ables two people to interact and engage …

Content truncated to 500 words