Evaluation of machine translation systems at CLS Corporate Language Services AG

Evaluation of machine translation systems at CLS Corporate Language Services AG Elisabeth Maier*, Anthony Clarke+, Hans-Udo Stadler+ + CLS Corporate Language Services AG Elisabethenanlage 11 CH-4051 Basel Switzerland anthony.clarke@cls.ch, hans-udo.stadler@cls.ch *Canoo Engineering AG Kirschgartenstr. 7 CH-4051 Basel Switzerland elisabeth.maier@canoo.com Abstract This paper describes the evaluation of Machine Translation (MT) System for use in a large company. To take into account the specific requirements of such an environment, a pragmatic approach for the evaluation was developed. It consists of five steps ranging from a specification of the evaluation process to the integration of the chosen MT system in a given infrastructure. The process includes a specification of MT evaluation criteria relevant to systems which have to be employed for a large customer base. The paper also shows the results of such an evaluation study which was recently carried out at CLS Corporate Language Services AG, where COMPRENDIUM is in the meantime being employed as corporate MT system. Keywords MT evaluation method, intelligibility, correctness, linguistic and non-linguistic evaluation criteria. Introduction Over the past few years, a growing number of large enterprises that operate on an international level have developed a growing need for a large amount of translations, which must be delivered in a very short time. The deployment of machine translation as a service offering to company employees and customers is the consequence. In the case of CLS Corporate Language Services AG, this need arose at its largest shareholder and customer, UBS AG, due to the multilinguality of UBS’s employees all over the world and especially in Switzerland. German and English have established themselves as core languages, while other languages such as French. Italian, Spanish etc. are becoming marginalized. Thus the need has arisen – at other multilingual companies as well – to have rapid, cost- effective and good quality translations of texts in German and English, which normally are not translated. In this context, Machine Translation is to be seen as a complement to human translation and not as a means to rationalise away human translators. The evaluation of Machine Translation (MT) systems for the deployment in large companies has a number of consequences on the evaluation method. Among the most important are: • The chosen system needs to comply with the technical standards of the company (e.g. system platform(s), availability, scalability); • The system must satisfy a range of business requirements (e.g. business languages and domains, types of use); • The evaluation process needs deliver fast results to guarantee rapid deployment. In the following chapter, we present a pragmatic approach for the evaluation of MT systems, which takes these factors into account. We first describe the evaluation process and then discuss the various evaluation criteria, which were used. In the subsequent chapter we show how this method was used for the evaluation of COMPRENDIUM, iTranslator, REVERSO and SYSTRAN, which was carried out in two stages at CLS Corporate Language Service AG. In conclusion, we describe the roadmap for the deployment of the system. Evaluation Process The evaluation process deployed …

Content truncated to 500 words