Improving Language Model Size Reduction using Better Pruning Criteria

Improving Language Model Size Reduction using Better Pruning Criteria Jianfeng Gao Microsoft Research, Asia Beijing, 100080, China jfgao@microsoft.com Min Zhang1 State Key Lab of Intelligent Tech & Sys. Computer Science & Technology Dept. Tsinghua University, China 1 This work was done while Zhang was working at Microsoft Research Asia as a visiting student. Abstract Reducing language model (LM) size is a critical issue when applying a LM to realistic applications which have memory constraints. In this paper, three measures are studied for the purpose of LM pruning. They are probability, rank, and entropy. We evaluated the performance of the three pruning criteria in a real application of Chinese text input in terms of character error rate (CER). We first present an empirical comparison, showing that rank performs the best in most cases. We also show that the high-performance of rank lies in its strong correlation with error rate. We then present a novel method of combining two criteria in model pruning. Experimental results show that the combined criterion consistently leads to smaller models than the models pruned using either of the criteria separately, at the same CER. 1 Introduction Backoff n-gram models for applications such as large vocabulary speech recognition are typically trained on very large text corpora. An uncompressed LM is usually too large for practical use since all realistic applications have memory constraints. Therefore, LM pruning techniques are used to produce the smallest model while keeping the performance loss as small as possible. Research on backoff n-gram model pruning has been focused on the development of the pruning criterion, which is used to estimate the performance loss of the pruned model. The traditional count cutoff method (Jelinek, 1990) used a pruning criterion based on absolute frequency while recent research has shown that better pruning criteria can be developed based on more sophisticated measures such as perplexity. In this paper, we study three measures for pruning backoff n-gram models. They are probability, rank and entropy. We evaluated the performance of the three pruning criteria in a real application of Chinese text input (Gao et al., 2002) through CER. We first present an empirical comparison, showing that rank performs the best in most cases. We also show that the high-performance of rank lies in its strong correlation with error rate. We then present a novel method of combining two pruning criteria in model pruning. Our results show that the combined criterion consistently leads to smaller models than the models pruned using either of the criteria separately. In particular, the combination of rank and entropy achieves the smallest models at a given CER. The rest of the paper is structured as follows: Section 2 discusses briefly the related work on backoff n-gram pruning. Section 3 describes in detail several pruning criteria. Section 4 presents an empirical comparison of pruning criteria using a Chinese text input system. Section 5 proposes our method of combining two criteria in model pruning. Section 6 presents conclusions and our future work. 2 Related Work N-gram models …

Content truncated to 500 words