An Information-Theoretic Empirical Analysis of Dependency-Based Feature Types for Word Prediction Models

An Information-Theoretic Empirical Analysis of Dependency-Based Feature Types for Word Prediction Models Dekai WU *, ZHAO Jun*, SUI Zhifang ** • 1" Human Language Technology Center Department of Computer Science University of Science & Technology, HKUST, Clear Water Bay, Hong Kong Computational Linguistics Institute Department of Computer Science & Technology Peking University, Beijing, 100871, P.R.China {dekai, zhaojun, Abstract Over the years, many proposals have been made to incorporate assorted types of feature in language models. However, discrepancies between training sets, evaluation criteria, algorithms, and hardware environments make it difficult to compare the models objectively. In this paper, we take an information theoretic approach to select feature types in a systematic manner. We describe a quantitative analysis of the information gain and the information redundancy for various combinations of feature types inspired by both dependency structure and bigram structure, using a Chinese treebank and taking word prediction as the object. The experiments yield several conclusions on the predictive value of several feature types and feature types combinations for word prediction, which are expected to provide guidelines for feature type selection in language modeling. 1 Introduction There are many types of features that a language model can use to predict a word in a sentence. Standard n-gram models use the immediately preceding words. Other fixed physical distance feature types may inspect word classes or parts of speech. Grammatically-based feature types may also be used, such as the suizf}@cs.ust.hk incident syntactic and semantic relations or the other words involved in those relations. Our ultimate aim is to determine which combination of feature types is optimal for language modeling. Unfortunately, the state of knowledge in this regard is very limited. Many language models have been published inspired by one or more of these feature types I11121131141151, but discrepancies between training sets, evaluation criteria, algorithms, and hardware environments make it difficult, if not impossible, to compare the models objectively. The paper uses an information theoretic approach to select feature types for language modeling in a systematic manner. We are concerned with quantitative analysis of the information quantity, information gain and the information redundancy for various feature type combinations in both dependency grammar structure and adjacent bigram structure. The experiments yield a number of conclusions on the predictive value of various feature types and the combinations thereof, which can provide useful information on what level of performance gain can be expected in principle from a bigram model augmented with long distance dependency features. The results are expected to provide a reliable reference for feature type selection in language modeling. We have used Chinese data for the experiments in this paper. Strictly speaking, our 138 I conclusions apply only to Chinese. However, we actually expect very similar results on English, and all our preliminary experiments on English data do bear this out I61. We believe the general methodology as well as many of the specific conclusions apply tO a wide range of languages. We will begin by introducing an information theoretic framework for feature type selection and …

Content truncated to 500 words