Evaluating Machine Translation in a Usage Scenario

Evaluating Machine Translation in a Usage Scenario Rosa Del Gaudio, Aljoscha Burchardt and Ant´onio Branco Higher Functions – Sistemas Inteligentes; DFKI – LT Lab; University of Lisbon Lisbon, Portugal; Berlin, Germany; Lisbon, Portugal rosa.gaudio@pcmedic.pt; aljoscha.burchardt@dfki.de; antonio.branco@di.fc.ul.pt Abstract In this document we report on a user-scenario-based evaluation aiming at assessing the performance of machine translation (MT) systems in a real context of use. We describe a sequel of experiments that has been performed to estimate the usefulness of MT and to test if improvements of MT technology lead to better performance in the usage scenario. One goal is to find the best methodology for evaluating the eventual benefit of a machine translation system in an application. The evaluation is based on the QTLeap corpus, a novel multilingual language resource that was collected through a real-life support service via chat. It is composed of naturally occurring utterances produced by users while interacting with a human technician providing answers. The corpus is available in eight different languages: Basque, Bulgarian, Czech, Dutch, English, German, Portuguese and Spanish. Keywords: Machine Translation, Evaluation, Multilingual Corpus 1. Introduction Extrinsic evaluation of MT, i.e. assessment of MT qual- ity impact within a task other than translation, has not (yet) been established as a major research topic. Reasons may in- clude the prevalent focus of MT research on translation of newspaper texts, which does not readily lend itself to task- based evaluation. In industrial applications of MT, task- based evaluation is certainly performed more frequently, but the results are typically not published. The evaluation reported in this paper joins together general MT research and industrially focused applications of MT. The goal is to find the best methodology for evaluating the eventual bene- fit of a machine translation system in a real-world applica- tion of the type considered here by resorting to a user-based scenario approach. This evaluation is based on the integration of MT services in a helpdesk application developed by the company Higher Functions as part of its business. It has been performed within the QTLeap project,1 which aims to investigate an articulated methodology for machine translation based on deep language engineering approaches. This paper is organised as follows. Section 2. provides some background information on the technical support sce- nario the evaluation is embedded in. Section 3. describes the QTLeap Corpus. Section 4. reports on the different ex- periments that constitute the evaluation. Finally, Section 5. concludes the paper. 2. Background: Technical Support Scenario The usage scenario adopted in our evaluation is based on a service developed by the company Higher Functions Lda. to support their clients. This service, named PCWizard, of- fers technical support by chat, through a call-centre. It tries to automate the process of answering simple and recurrent user requests for IT troubleshooting for both hardware and software. This problem solving procedure has been made efficient by using a Question Answering (QA) application and repos- 1www.qtleap.eu Figure 1: The workflow with the MT services itory that helps call-centre operators prepare replies for clients. Using …

Content truncated to 500 words