Evaluating the Learning Curve of Domain Adaptive Statistical Machine Translation Systems

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 433–441, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Evaluating the Learning Curve of Domain Adaptive Statistical Machine Translation Systems Nicola Bertoldi Mauro Cettolo Marcello Federico Fondazione Bruno Kessler via Sommarive 18 38123 Trento, Italy <surename>@fbk.eu Christian Buck University of Edinburgh 10 Crichton Street EH8 9AB Edinburgh, UK christian.buck@ed.ac.uk Abstract The new frontier of computer assisted transla- tion technology is the effective integration of statistical MT within the translation workflow. In this respect, the SMT ability of incremen- tally learning from the translations produced by users plays a central role. A still open problem is the evaluation of SMT systems that evolve over time. In this paper, we propose a new metric for assessing the quality of an adaptive MT component that is derived from the theory of learning curves: the percentage slope. 1 Introduction Translation memories and computer assisted trans- lation (CAT) tools are currently the dominant tech- nologies in the translation and localization market, but recent achievements in statistical MT have raised new expectations in the translation industry. So far, statistical MT has focused on providing ready-to-use translations, rather than outputs that minimize the effort of a human translator. The MateCAT project1 aims at pushing what can be considered the new frontier of CAT technology: how to effectively inte- grate statistical MT within the translation workflow. One pursued research direction is developing do- main adaptive SMT models, i.e. models that dynam- ically adapt to the translations that are continuously added to the translation memory by the user dur- ing her/his work. The ideal goal is to progressively reduce the mismatch between training and testing 1http://www.matecat.com/ data, in such a way that the adapted SMT engine will be able to provide the user with useful suggestions – i.e. perfect or worth being post-edited – when the translation memory fails to retrieve perfect or almost perfect matches. Among the well known machine learning paradigms that fit with this scenario are on- line learning and incremental learning, which basi- cally differ in the amount of data that is employed to dynamically adapt the system: a single piece of data in the first case and a batch of data in the lat- ter. Notice that in both cases one assumes that do- main adaptation is performed efficiently, i.e. by only processing the newly received data. Moreover, al- though the quantity of acquired in-domain data is generally limited, their high quality and relevance to the translation task justify their exploitation by all means possible. Domain adaptive SMT embeds two challenges: (1) the design of effective adaptation algorithms, and (2) the evaluation of MT systems evolving over time. Since the ultimate goal of our efforts is to increase the productivity of human translators, the most ac- curate assessment methodology would be of course to run a field test. This way, we could compare pro- ductivity of human translators receiving suggestions from an MT engine featuring dynamic domain adap- tation against the productivity …

Content truncated to 500 words