Measuring Machine Translation Errors in New Domains

Measuring Machine Translation Errors in New Domains Ann Irvine Johns Hopkins University anni@jhu.edu John Morgan University of Maryland jjm@cs.umd.edu Marine Carpuat National Research Council Canada marine.carpuat@nrc.gc.ca Hal Daum´e III University of Maryland me@hal3.name Dragos Munteanu SDL Research dmunteanu@sdl.com Abstract We develop two techniques for analyzing the effect of porting a machine translation system to a new domain. One is a macro-level ana- lysis that measures how domain shift affects corpus-level evaluation; the second is a micro- level analysis for word-level errors. We ap- ply these methods to understand what happens when a Parliament-trained phrase-based ma- chine translation system is applied in four very different domains: news, medical texts, scien- tific articles and movie subtitles. We present quantitative and qualitative experiments that highlight opportunities for future research in domain adaptation for machine translation. 1 Introduction When building a statistical machine translation (SMT) system, the expected use case is often limited to a specific domain, genre and register (henceforth “domain” refers to this set, in keeping with standard, imprecise, terminology), such as a particular type of legal or medical document. Unfortunately, it is ex- pensive to obtain enough parallel data to reliably es- timate translation models in a new domain. Instead, one can hope that large amounts of data from ano- ther, “old domain,” might be close enough to stand as a proxy. This is the defacto standard: we train SMT systems on Parliament proceedings, but then use them to translate all sorts of new text. Unfortuna- tely, this results in significantly degraded translation quality. In this paper, we present two complemen- tary methods for quantifiably measuring the source of translation errors (§5.1 and §5.2) in a novel taxo- nomy (§4). We show quantitative (§7.1) and quali- tative (§7.2) results obtained from our methods on Old Domain (Hansard) Inp monsieur le pr´esident, les pˆecheurs de homard de la r´egion de l’atlantique sont dans une situation catastro- phique. Ref mr. speaker, lobster fishers in atlantic canada are facing a disaster. Out mr. speaker, the lobster fishers in atlantic canada are in a mess. New Domain (Medical) Inp mode et voie(s) d’administration Ref method and route(s) of administration Out fashion and voie(s) of directors TABLE 1: Example inputs, references and system outputs. There are three types of errors: unseen words (blue), in- correct sense selection (red) and unknown sense (green). four very different new domains: newswire, medical texts, scientific abstracts, and movie subtitles. Our basic approach is to think of translation er- rors in the context of a novel taxonomy of error categories, “S4.” Our taxonomy contains categories for the errors shown in Table 1, in which an SMT system trained on the Hansard parliamentary proce- dings is applied to a new domain (in this case, me- dical texts). Our categorization focuses on the follo- wing: new French words, new French senses, and in- correctly chosen translations. The first methodology we develop for studying such errors is a micro-level study of the frequency and distribution of these error types in real translation output at …

Content truncated to 500 words