A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English

MT Summit VII Sept. 1999 A New Evaluation Method for Speech Translation Systems and a Case Study on ATR-MATRIX from Japanese to English Toshiyuki Takezawa Fumiaki Sugaya ATR Interpreting Telecommunications ATR Interpreting Telecommunications Research Laboratories. Kyoto, Japan Research Laboratories. Kyoto. Japan Akio Yokoo Seiichi Yamamoto ATR Interpreting Telecommunications ATR Interpreting Telecommunications Research Laboratories. Kyoto. Japan Research Laboratories. Kyoto. Japan Abstract ATR-MATRIX is a multi-lingual speech- to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel ar- rangement domain. In this paper, we pro- pose a new evaluation method for speech translation systems. Our current focus is on measuring the robustness of a language translation sub-system, with quick calcula- tion and low cost. Therefore, we calculate the difference between the translation out- put from transcription texts and the trans- lation output from input speech by a dy- namic programming method. We present the first trial experiment of this method applied to our Japanese-to-English speech translation system. We also provide re- lated discussions on such points as error analysis and the relationship between the proposed method and translation quality evaluation manually done by humans. 1 Introduction In the coming 21st century, the global demand for communications between speakers of different lan- guages is expected to grow. According to recent sur- veys [1,2],automatic speech translation will proba- bly come into practical use for the general public by around 2010-2020. The ideal speech translation sys- tem is expected to automatically and instantly con- vey the speaker's ideas in the language of the listener. Such a system is also expected to allow the speaker to speak naturally without having to give any special consideration to the translation system. However, the expectations of such an ideal speech translation sys- tem are still hampered by a mountain of technical dif- ficulties. NEC conducted a demonstration of a prototype speech translation system at the Telecom '83 exhibi- tion. ATR Interpreting Telephony Research Labora- tories was established in 1986 to conduct research on basic technologies for speech translation. ASURA [3] is one of the research results of ATR Interpreting Tele- phony Research Laboratories. This system can recog- nize well-formed Japanese utterances in a limited do- main, translate them into both English and German, and output synthesized speech. The ASURA system was used for the International Joint Experiment on Interpreting Telephony with participants from Kyoto. Japan (ATR). Pittsburgh. USA (Carnegie Mellon Uni- versity [4]), and Munich, Germany (Siemens and the University of Karlsruhe) in January 1993 [3]. Many projects on speech-to-speech translation be- gan at that time [5, 6, 7]. SRI International and Swedish Telecom developed a prototype speech trans- lation system that could translate queries from spoken English to spoken Swedish in the domain of air travel information systems [5]. AT&T Bell Laboratories and Telefonica Investigacion y Desarrollo developed a re- stricted domain spoken language translation system called VEST (Voice English/Spanish Translator) [6]. In Germany. Verbmobil [7] was created as a major speech-to-speech translation research project. The Verbmobil …

Content truncated to 500 words