Learning Performance of a Machine Translation System: a Statistical and Computational Analysis

Proceedings of the Third Workshop on Statistical Machine Translation, pages 35–43, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Learning Performance of a Machine Translation System: a Statistical and Computational Analysis Marco Turchi Tijl De Bie Dept. of Engineering Mathematics University of Bristol, Bristol, BS8 1TR, UK {Marco.Turchi, Tijl.DeBie}@bristol.ac.uk nello@support-vector.net Nello Cristianini Abstract We present an extensive experimental study of a Statistical Machine Translation system, Moses (Koehn et al., 2007), from the point of view of its learning capabilities. Very ac- curate learning curves are obtained, by us- ing high-performance computing, and extrap- olations are provided of the projected perfor- mance of the system under different condi- tions. We provide a discussion of learning curves, and we suggest that: 1) the represen- tation power of the system is not currently a limitation to its performance, 2) the inference of its models from finite sets of i.i.d. data is responsible for current performance limita- tions, 3) it is unlikely that increasing dataset sizes will result in significant improvements (at least in traditional i.i.d. setting), 4) it is un- likely that novel statistical estimation methods will result in significant improvements. The current performance wall is mostly a conse- quence of Zipf’s law, and this should be taken into account when designing a statistical ma- chine translation system. A few possible re- search directions are discussed as a result of this investigation, most notably the integra- tion of linguistic rules into the model inference phase, and the development of active learning procedures. 1 Introduction and Background The performance of every learning system is the re- sult of (at least) two combined effects: the repre- sentation power of the hypothesis class, determin- ing how well the system can approximate the target behaviour; and statistical effects, determining how well the system can approximate the best element of the hypothesis class, based on finite and noisy train- ing information. The two effects interact, with richer classes being better approximators of the target be- haviour but requiring more training data to reliably identify the best hypothesis. The resulting trade- off, equally well known in statistics and in machine learning, can be expressed in terms of bias variance, capacity-control, or model selection. Various theo- ries on learning curves have been proposed to deal with it, where a learning curve is a plot describing performance as a function of some parameters, typ- ically training set size. In the context of Statistical Machine Translation (SMT), where large bilingual corpora are used to train adaptive software to translate text, this task is further complicated by the peculiar distribution un- derlying the data, where the probability of encoun- tering new words or expressions never vanishes. If we want to understand the potential and limitations of the current technology, we need to understand the interplay between these two factors affecting perfor- mance. In an age where the creation of intelligent behaviour is increasingly data driven, this is a ques- tion of great importance to all of Artificial Intelli- gence. These observations …

Content truncated to 500 words