Toward an Interagency Language Roundtable Based Assessment of Speech-to-Speech Translation Capabilities

Toward an Interagency Language Roundtable Based Assessment of Speech-to-Speech Translation Capabilities Douglas Jones, Wade Shen, Brian Delaney, Martha Herzog, Michael Emonts, Sabine Atwell , James Dirgin, Neil Granoien, Sargon Jabri, Jurgen Sottung, MIT Lincoln Laboratory DLI Foreign Language Center Lexington, MA 02420, USA Monterey, CA 93944, USA {DAJ,BDelaney,SWade}LL.MIT.edu MHerzog2005@comcast.net* {Michael.Emonts,Sabine.Atwell,James.Dirgin, Sargon.Jabri,Jurgen.Sottung} @monterey.army.mil; granoien@sbcglobal.net* Timothy Anderson, Timothy Hunter Air Force Research Laboratory United States Army Intelligence Center Wright Patterson AFB, OH 45433, USA Ft. Huachuca, AZ 85613, USA Timothy.Anderson@wpafb.af.mil Timmie.Hunter@us.army.mil Abstract• We present observations from three exer- cises designed to map the effective listen- ing and speaking skills of an operator of a speech-to-speech translation system (S2S) to the Interagency Language Roundtable (ILR) scale. Such a mapping is non- trivial, but will be useful for government and military decision makers in managing expectations of S2S technology. We ob- served domain-dependent S2S capabilities in the ILR range of Level 0+ to Level 1, and interactive text-based machine trans- lation in the Level 3 range. 1 Introduction We present observations of three exercises that we conducted at a three-day workshop in September • This work is sponsored by the Defense Language Institute under Air Force contract FA8721-05-C-0002. Opinions, inter- pretations, conclusions and recommendations are those of the authors and are not necessarily endorsed by the United States Government. * Dr. Herzog is working under Research Subcontract with MIT Lincoln Laboratory. * Dr. Granoien retired from DLI in April 2006 and continues to be active in language learning research. 2005 at the Defense Language Institute Foreign Language Center in Monterey, California. The purpose of this workshop was to identify which ILR based testing methods might be suitable for adaptation to S2S evaluation. The U.S. Depart- ment of Defense uses ILR-based testing for high- stakes tests that assess foreign language skills on the part of human language learners. Being able to characterize S2S technology in terms of the ILR will be useful for government and military decision makers in relating current S2S capabilities and limitations to a known measure. As expected, we learned that the current methods of testing foreign language learners need to modified, in some cases quite substantially. The primary difference for S2S evaluation is that domain dependence required us to make adjustments. Translation errors were also a factor. Our starting point for measuring S2S capabili- ties was the Oral Proficiency Interview (OPI), a standardized test of speaking skills that is also able to test listening skills. However, the OPI is de- signed as a general language proficiency test, not a domain-specific or job-related performance test. Our focus here is on a job-related test, since S2S systems are currently domain-dependent. Thus we needed to create a more specialized test, which we are calling the Job-related Speaking Performance Test (JSPT) that can also be used to test listening skills. 82 Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pages 82-89, Cambridge, August 2006. ©2006 The Association for Machine Translation in the Americas We first had to address several …

Content truncated to 500 words