An Information-Theory-Based Feature Type Analysis for the Modeling of Statistical Parsing

An Information-Theory-Based Feature Type Analysis for the Modelling of Statistical Parsing SUI Zhifang †‡, ZHAO Jun †, Dekai WU † † Hong Kong University of Science & Technology Department of Computer Science Human Language Technology Center Clear Water Bay, Hong Kong ‡ Peking University Department of Computer Science & Technology Institute of Computational Linguistics Beijing, China suizf@icl.pku.edu.cn, zhaojun@cs.ust.hk, dekai@cs.ust.hk Abstract The paper proposes an information-theory- based method for feature types analysis in probabilistic evaluation modelling for statistical parsing. The basic idea is that we use entropy and conditional entropy to measure whether a feature type grasps some of the information for syntactic structure prediction. Our experiment quantitatively analyzes several feature types’ power for syntactic structure prediction and draws a series of interesting conclusions. 1 Introduction In the field of statistical parsing, various probabilistic evaluation models have been proposed where different models use different feature types [Black, 1992] [Briscoe, 1993] [Brown, 1991] [Charniak, 1997] [Collins, 1996] [Collins, 1997] [Magerman, 1991] [Magerman, 1992] [Magerman, 1995] [Eisner, 1996]. How to evaluate the different feature types’ effects for syntactic parsing? The paper proposes an information-theory-based feature types analysis model, which uses the measures of predictive information quantity, predictive information gain, predictive information redundancy and predictive information summation quantitatively analyse the different contextualto feature types’ or feature types combination’s predictive power for syntactic structure. In the following, Section 2 describes the probabilistic evaluation model for syntactic trees; Section 3 proposes an information-theory-based feature type analysis model; Section introduces several experimental issues; Section 54 quantitatively analyses the different contextual feature types or feature types combination in the view of information theory and draws a series of conclusion on their predictive powers for syntactic structures. 2 The probabilistic evaluation model for statistical syntactic parsing Given a sentence, the task of statistical syntactic parsing is to assign a probability to each candidate parsing tree that conforms to the grammar and select the one with highest probability as the final analysis result. That is: ) | argmax( PTS T T best = (1) where S denotes the given sentence, T denotes the set of all the candidate parsing trees that conform to the grammar, P(T|S) denotes the probability of parsing tree T for the given sentence S. The task of probabilistic evaluation model in syntactic parsing is the estimation of P(T|S). In the syntactic parsing model which uses rule- based grammar, the probability of a parsing tree can be defined as the probability of the derivation which generates the current parsing tree for the given sentence. That is, ∏ ∏ = = − = = = n i i i n i i i n S h r P S r rr r P S r Prr TS P 1 1 1 2 1 2 1 ,) | ( ,) , , , | ( ) | , , (, ) | ( � � (2) Where, 1 2 1 , , , ir− rr � denotes a derivation rule sequence, hi denotes the partial parsing tree derived from …

Content truncated to 500 words