How Many Bits Are Needed To Store Probabilities for Phrase-Based Translation?

Proceedings of the Workshop on Statistical Machine Translation, pages 94–101, New York City, June 2006. c⃝2006 Association for Computational Linguistics How Many Bits Are Needed To Store Probabilities for Phrase-Based Translation? Marcello Federico and Nicola Bertoldi ITC-irst - Centro per la Ricerca Scientifica e Tecnologica 38050 Povo - Trento, Italy {federico,bertoldi}@itc.it Abstract State of the art in statistical machine trans- lation is currently represented by phrase- based models, which typically incorpo- rate a large number of probabilities of phrase-pairs and word n-grams. In this work, we investigate data compression methods for efficiently encoding n-gram and phrase-pair probabilities, that are usu- ally encoded in 32-bit floating point num- bers. We measured the impact of com- pression on translation quality through a phrase-based decoder trained on two dis- tinct tasks: the translation of European Parliament speeches from Spanish to En- glish, and the translation of news agencies from Chinese to English. We show that with a very simple quantization scheme all probabilities can be encoded in just 4 bits with a relative loss in BLEU score on the two tasks by 1.0% and 1.6%, respectively. 1 Introduction In several natural language processing tasks, such as automatic speech recognition and machine transla- tion, state-of-the-art systems rely on the statistical approach. Statistical machine translation (SMT) is based on parametric models incorporating a large num- ber of observations and probabilities estimated from monolingual and parallel texts. The current state of the art is represented by the so-called phrase-based translation approach (Och and Ney, 2004; Koehn et al., 2003). Its core components are a translation model that contains probabilities of phrase-pairs, and a language model that incorporates probabilities of word n-grams. Due to the intrinsic data-sparseness of language corpora, the set of observations increases almost lin- early with the size of the training data. Hence, to efficiently store observations and probabilities in a computer memory the following approaches can be tackled: designing compact data-structures, pruning rare or unreliable observations, and applying data compression. In this paper we only focus on the last approach. We investigate two different quantization methods to encode probabilities and analyze their impact on translation performance. In particular, we address the following questions: • How does probability quantization impact on the components of the translation system, namely the language model and the translation model? • Which is the optimal trade-off between data compression and translation performance? • How do quantized models perform under dif- ferent data-sparseness conditions? • Is the impact of quantization consistent across different translation tasks? Experiments were performed with our phrase- based SMT system (Federico and Bertoldi, 2005) on two large-vocabulary tasks: the translation of Euro- pean Parliament Plenary Sessions from Spanish to 94 English, and the translation of news agencies from Chinese to English, according to the set up defined by the 2005 NIST MT Evaluation Workshop. The paper is organized as follows. Section 2 re- views previous work addressing efficiency in speech recognition and information retrieval. Section 3 in- troduces the two quantization methods considered in this paper, …

Content truncated to 500 words