Stream-based Randomised Language Models for SMT

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 756–764, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Stream-based Randomised Language Models for SMT Abby Levenberg School of Informatics University of Edinburgh a.levenberg@sms.ed.ac.uk Miles Osborne School of Informatics University of Edinburgh miles@inf.ed.ac.uk Abstract Randomised techniques allow very big language models to be represented suc- cinctly. However, being batch-based they are unsuitable for modelling an un- bounded stream of language whilst main- taining a constant error rate. We present a novel randomised language model which uses an online perfect hash function to efficiently deal with unbounded text streams. Translation experiments over a text stream show that our online ran- domised model matches the performance of batch-based LMs without incurring the computational overhead associated with full retraining. This opens up the possibil- ity of randomised language models which continuously adapt to the massive volumes of texts published on the Web each day. 1 Introduction Language models (LM) are an integral feature of statistical machine translation (SMT) systems. They assign probabilities to generated hypothe- ses in the target language informing lexical selec- tion. The most common form of LMs in SMT systems are smoothed n-gram models which pre- dict a word based on a contextual history of n − 1 words. For some languages (such as English) tril- lions of words are available for training purposes. This fact, along with the observation that ma- chine translation quality improves as the amount of monolingual training material increases, has lead to the introduction of randomised techniques for representing large LMs in small space (Talbot and Osborne, 2007; Talbot and Brants, 2008). Randomised LMs (RLMs) solve the problem of representing large, static LMs but they are batch oriented and cannot incorporate new data with- out fully retraining from scratch. This property makes current RLMs ill-suited for modelling the massive volume of textual material published daily on the Web. We present a novel RLM which is capable of incremental (re)training. We use ran- dom hash functions coupled with an online perfect hashing algorithm to represent n-grams in small space. This makes it well-suited for dealing with an unbounded stream of training material. To our knowledge this is the first stream-based RLM re- ported in the machine translation literature. As well as introducing the basic stream-based RLM, we also consider adaptation strategies. Perplex- ity and translation results show that populating the language model with material chronologically close to test points yields good results. As with previous randomised language models, our experi- ments focus on machine translation but we also ex- pect that our findings are general and should help inform the design of other stream-based models. Section 2 introduces the incrementally retrain- able randomised LM and section 3 considers re- lated work; Section 4 then considers the question of how unbounded text streams should be mod- elled. Sections 5 and 6 show stream-based trans- lation results and properties of our novel data- structure. Section 7 concludes the paper. 2 Online Bloomier Filter LM Our …

Content truncated to 500 words