Multiple-stream Language Models for Statistical Machine Translation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 177–186, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Multiple-stream Language Models for Statistical Machine Translation Abby Levenberg Dept. of Computer Science University of Oxford ablev@cs.ox.ac.uk Miles Osborne School of Informatics University of Edinburgh miles@inf.ed.ac.uk David Matthews School of Informatics University of Edinburgh dave.matthews@ed.ac.uk Abstract We consider using online language models for translating multiple streams which naturally arise on the Web. After establishing that us- ing just one stream can degrade translations on different domains, we present a series of simple approaches which tackle the problem of maintaining translation performance on all streams in small space. By exploiting the dif- fering throughputs of each stream and how the decoder translates prior test points from each stream, we show how translation perfor- mance can equal specialised, per-stream lan- guage models, but do this in a single language model using far less space. Our results hold even when adding three billion tokens of addi- tional text as a background language model. 1 Introduction There is more natural language data available today than there has ever been and the scale of its produc- tion is increasing quickly. While this phenomenon provides the Statistic Machine Translation (SMT) community with a potentially extremely useful re- source to learn from, it also brings with it nontrivial computational challenges of scalability. Text streams arise naturally on the Web where millions of new documents are published each day in many different languages. Examples in the stream- ing domain include the thousands of multilingual websites that continuously publish newswire stories, the official proceedings of governments and other bureaucratic organisations, as well as the millions of “bloggers” and host of users on social network services such as Facebook and Twitter. Recent work has shown good results using an in- coming text stream as training data for either a static or online language model (LM) in an SMT setting (Goyal et al., 2009; Levenberg and Osborne, 2009). A drawback of prior work is the oversimplified sce- nario that all training and test data is drawn from the same distribution using a single, in-domain stream. In a real world scenario multiple incoming streams are readily available and test sets from dissimilar do- mains will be translated continuously. As we show, using stream data from one domain to translate an- other results in poor average performance for both streams. However, combining streams naively to- gether hurts performance further still. In this paper we consider this problem of multiple stream translation. Since monolingual data is very abundant, we focus on the subtask of updating an on- line LM using multiple incoming streams. The chal- lenges in multiple stream translation include dealing with domain differences, variable throughput rates (the size of each stream per epoch), and the need to maintain constant space. Importantly, we impose the key requirement that our model match transla- tion performance reached using the single stream ap- proach on all test domains. We accomplish this using the n-gram …

Content truncated to 500 words