Stream-based Translation Models for Statistical Machine Translation

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 394–402, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Stream-based Translation Models for Statistical Machine Translation Abby Levenberg School of Informatics University of Edinburgh a.levenberg@ed.ac.uk Chris Callison-Burch Computer Science Department Johns Hopkins University ccb@cs.jhu.edu Miles Osborne School of Informatics University of Edinburgh miles@inf.ed.ac.uk Abstract Typical statistical machine translation sys- tems are trained with static parallel corpora. Here we account for scenarios with a continu- ous incoming stream of parallel training data. Such scenarios include daily governmental proceedings, sustained output from transla- tion agencies, or crowd-sourced translations. We show incorporating recent sentence pairs from the stream improves performance com- pared with a static baseline. Since frequent batch retraining is computationally demand- ing we introduce a fast incremental alternative using an online version of the EM algorithm. To bound our memory requirements we use a novel data-structure and associated training regime. When compared to frequent batch re- training, our online time and space-bounded model achieves the same performance with significantly less computational overhead. 1 Introduction There is more parallel training data available to- day than there has ever been and it keeps increas- ing. For example, the European Parliament1 releases new parallel data in 22 languages on a regular basis. Project Syndicate2 translates editorials into seven languages (including Arabic, Chinese and Russian) every day. Existing translation systems often get ‘crowd-sourced’ improvements such as the option to contribute a better translation to GoogleTrans- late3. In these and many other instances, the data can be viewed as an incoming unbounded stream since 1http://www.europarl.europa.eu 2http://www.project-syndicate.org 3http://www.translate.google.com the corpus grows continually with time. Dealing with such unbounded streams of parallel sentences presents two challenges: making retraining efficient and operating within a bounded amount of space. Statistical Machine Translation (SMT) systems are typically batch trained, often taking many CPU- days of computation when using large volumes of training material. Incorporating new data into these models forces us to retrain from scratch. Clearly, this makes rapidly adding newly translated sen- tences into our models a daunting engineering chal- lenge. We introduce an adaptive training regime us- ing an online variant of EM that is capable of in- crementally adding new parallel sentences without incurring the burdens of full retraining. For situations with large volumes of incoming parallel sentences we are also forced to consider placing space-bounds on our SMT system. We in- troduce a dynamic suffix array which allows us to add and delete parallel sentences, thereby maintain- ing bounded space despite processing a potentially high-rate input stream of unbounded length. Taken as a whole we show that online translation models operating within bounded space can perform as well as systems which are batch-based and have no space constraints thereby making our approach suitable for stream-based translation. 2 Stepwise Online EM The EM algorithm is a common way of inducing latent structure from unlabeled data in an unsuper- vised manner (Dempster et al., 1977). Given a set of unlabeled examples …

Content truncated to 500 words