Probing the Lexicon in Evaluating Commercial MT Systems

Probing the lexicon in evaluating commercial MT systems Martin Volk University of Zurich Department of Computer Science, Computational Linguistics Group Winterthurerstr. 190, CH-8057 Zurich volk©ifi, unizh, ch Abstract In the past the evaluation of machine trans- lation systems has focused on single sys- tem evaluations because there were only few systems available. But now there are several commercial systems for the same language pair. This requires new methods of comparative evaluation. In the paper we propose a black-box method for comparing the lexical coverage of MT systems. The method is based on lists of words from dif- ferent frequency classes. It is shown how these word lists can be compiled and used for testing. We also present the results of using our method on 6 MT systems that translate between English and German. 1 Introduction The evaluation of machine translation (MT) sys- tems has been a central research topic in recent years (cp. (Sparck-Jones and Galliers, 1995; King, 1996)). Many suggestions have focussed on measur- ing the translation quality (e.g. error classification in (Flanagan, 1994) or post editing time in (Minnis, 1994)). These measures are time-consuming and dif- ficult to apply. But translation quality rests on the linguistic competence of the MT system which again is based first and foremost on grammatical coverage and lexicon size. Testing grammatical coverage can be done by using a test suite (cp. (Nerbonne et al., 1993; Volk, 1995)). Here we will advocate a prob- ing method for determining the lexical coverage of commercial MT systems. We have evaluated 6 MT systems which translate between English and German and which are all po- sitioned in the low price market (under US$ 1500). • German Assistant in Accent Duo V2.0 (de- veloper: MicroTac/Globalink; distributor: Ac- cent) * Langenscheidts T1 Standard V3.0 (developer: GMS; distributor: Langenscheidt) • Personal Translator plus V2.0 (developer: IBM; distributor: von Rheinbaben & Busch) • Power Translator Professional (developer/dis- tributor: Globalink) 1 • Systran Professional for Windows (developer: Systran S.A.; distributor: Mysoft) • Telegraph V1.0 (developer/distributor: Glob- alink) The overall goal of our evaluation was a compar- ison of these systems resulting in recommendations on which system to apply for which purpose. The evaluation consisted of compiling a list of criteria for self evaluation and three experiments with ex- ternal volunteers, mostly students from a local in- terpreter school. These experiments were performed to judge the information content of the translations, the translation quality, and the user-friendliness. The list of criteria for self evaluation consisted of technical, linguistic and ergonomic issues. As part of the linguistic evaluation we wanted to determine the lexical coverage of the MT systems since only some of the systems provide figures on lexicon size in the documentation. Many MT system evaluations in the past have been white-box evaluations performed by a test- ing team in cooperation with the developers (see (Falkedal, 1991) for a survey). But commercial MT systems can only be evaluated in a black-box setup since the developer typically will not make the source code and …

Content truncated to 500 words