Towards fairer evaluations of commercial MT systems on basic travel expressions corpora

Towards Fairer Evaluations of Commercial MT Systems on Basic Travel Expressions Corpora Hervé Blanchon†, Christian Boitet†, Francis Brunet-Manquat†, Mutsuko Tomokiyo†, Agnès Hamon‡, Vo Trung Hung†, Youcef Bey† † Laboratoire CLIPS BP 53 38041 Grenoble Cedex 9, France {first.last}@imag.fr ‡ Laboratoire de Statistique (Université Haute Bretagne) Place du recteur H. Le Moal, CS 24307 35043 Rennes Cedex, France Agnès.Hamon@uhb.fr Abstract We compare the performance of several SYSTRAN systems on the BTEC corpus. Two language pairs: Chinese to English and Japanese to English are used. Whenever it is possible the system will be used “off the shelf” and then tuned. The first system we use is freely available on the web. The second system, SYSTRAN Premium, is commercial. It is used in two ways: (1) choosing and ordering available original dictionaries and setting parameters, (2) same + user dictionaries. As far as the evaluation is concerned, we competed in the unlimited data track. 1. Introduction We first give our motivations for participating in this campaign with a commercial system. Then we briefly describe the system tested (SYSTRAN® 5.0) and the ways it can be parametrized. The bulk of this paper describes the evaluation procedure. We finish by presenting and analyzing the results. 2. Rationale MT evaluation is a hot topic since 1960 or so. The literature on evaluation may even be larger than that on MT techniques proper. MT evaluation may have several goals: – help buyers buy MT (or CAT) system best suited to their needs, – help funders decide on which technology to support, and – help developers measure various aspects of their systems, and measure progress. The MT evaluation campaign organized by the C-STAR III consortium falls in the latter category. Its aim is to measure the “quality” of various MT systems developed for speech-to- speech translation when applied to the BTEC corpus [7]. Another goal is to compare the MT systems developed by the C-STAR partner not only between them, but also with other systems, notably commercial systems. In the past, we often got the impression that, in similar campaigns, the commercial system used as a “baseline” were tested in quite biased ways. From what was reported, the experimenters submitted the input texts to free MT web servers, and evaluated the results. But that method is quite unfair, which makes all the conclusions scientifically invalid. For example, long ago, the CANDIDE system trained intensively on the Hansard corpus, was compared with an off- the-shelf version of SYTSRAN without any tuning. SYSTRAN clearly won, but the margin might have been far bigger (or perhaps not, this should have been studied!), if SYSTRAN had been tuned to this totally unseen corpus, at the level of dictionaries, of course, but perhaps also of grammars. Another example is given by MSR [5] on comparison between their French-English system, highly tuned to their documents (actually, the transfer component was 100% induced from 150000 pairs of sentences and their associated “logical forms” or deep syntactic trees). They used also SYSTRAN, this time slightly tuning …

Content truncated to 500 words