A Phrase Table without Phrases: Rank Encoding for Better Phrase Table Compression

A Phrase Table without Phrases: Rank Encoding for Better Phrase Table Compression Marcin Junczys-Dowmunt Faculty of Mathematics and Computer Science Adam Mickiewicz University, Pozna´n, Poland junczys@amu.edu.pl Abstract This paper describes the first steps towards a minimum-size phrase table implemen- tation to be used for phrase-based statis- tical machine translation. The focus lies on the size reduction of target language data in a phrase table. Rank Encoding (R- Enc), a novel method for the compression of word-aligned target language in phrase tables is presented. Combined with Huff- man coding a relative size reduction of 56 percent for target phrase words and align- ment data is achieved when compared to bare Huffman coding without R-Enc. In the context of the complete phrase table the size reduction is 22 percent. 1 Introduction As the size of available parallel corpora increases, the size of translation phrase tables used for sta- tistical machine translation extracted from these corpora increases even faster. Although phrase ta- ble filtering methods (Johnson et al., 2007) have been described and physical memory as well as disk space are cheap, even current high-end sys- tems can be pushed to their limits. The current in- memory representation of a phrase-table in Moses (Koehn et al., 2007), a widely used open-source statistical machine toolkit, is unusable for anything else but toy-size translation models or prefiltered test set data. A binary on-disk implementation of a phrase table is generally used, but its on-disk size requirements are significant.1 The goal of this paper is to describe the first steps towards a compact phrase table implementa- c⃝ 2012 European Association for Machine Translation. 1The need for a more compact phrase-table implementation arose during the author’s collaboration with the MT team at WIPO. The space requirements of the binary representations of the phrase table and the reordering table for a single lan- guage pair exceeded the space available on a single SSD hard drive. tion that can be used as a drop-in replacement for both, the binary phrase table implementation and the in-memory phrase table available in Moses. An important requirement is the faithful produc- tion of translations identical to translations gener- ated from the original phrase table implementation if the same settings are provided. The general idea is to trade in processor time for disk and memory space. Instead of keeping fully constructed target phrases in the phrase table, they are stored as Huffman compressed sequences of bytes. On demand, they are decompressed, de- coded, and constructed as objects during run-time. As we show later, this does not necessarily mean that performance is negatively affected. Even better compression can be achieved with a dedicated encoding method of target words de- veloped for translation phrase tables. Rank En- coding (R-Enc) exploits the fact that target phrase words can be reduced to abstract symbols that de- scribe properties of source phrase words rather than target words. The statistical distribution of these abstract symbols in the phrase table allows for a much better choice of Huffman codes. 2 Related …

Content truncated to 500 words