Independence Assumptions Considered Harmful

Independence Assumptions Considered Harmful Alexander Franz Sony Computer Science Laboratory &: D21 Laboratory Sony Corporation 6-7-35 Kitashinagawa Shinagawa-ku, Tokyo 141, Japan amI©csl, sony. co. jp Abstract Many current approaches to statistical lan- guage modeling rely on independence a.~- sumptions 1)etween the different explana- tory variables. This results in models which are computationally simple, but which only model the main effects of the explanatory variables oil the response vari- able. This paper presents an argmnent in favor of a statistical approach that also models the interactions between the ex- planatory variables. The argument rests on empirical evidence from two series of ex- periments concerning automatic ambiguity resolution. 1 Introduction In this paper, we present an empirical argument in favor of a certain approach to statistical natural lan- guage modeling: we advocate statistical natural lan- guage models that account for the interactions be- tween the explanatory statistical variables, rather than relying on independence a~ssumptions. Such models are able to perform prediction on the basis of estimated probability distributions that are properly conditioned on the combinations of the individual values of the explanatory variables. After describing one type of statistical model that is particularly well-suited to modeling natural lan- guage data, called a loglinear model, we present ein- pirical evidence fi'om a series of experiments on dif- ferent ambiguity resolution tasks that show that the performance of the loglinear models outranks the performance of other models described in the lit- erature that a~ssume independence between the ex- planatory variables. 2 Statistical Language Modeling By "statistical language model", we refer to a mathe- matical object that "imitates the properties" of some respects of naturM language, and in turn makes pre- dictions that are useful from a scientific or engineer- ing point of view. Much recent work in this flame- work hm~ used written and spoken natural language data to estimate parameters for statisticM models that were characterized by serious limitations: mod- els were either limited to a single explanatory vari- able or. if more than one explanatory variable wa~s considered, the variables were assumed to be inde- pendent. In this section, we describe a method for statistical language modeling that transcends these limitations. 2.1 Categorical Data Analysis Categorical data analysis is the area of statistics that addresses categorical statistical variable: variables whose values are one of a set of categories. An exam- pie of such a linguistic variable is PART-OF-SPEECH, whose possible values might include nou.n, verb, de- terminer, preposition, etc. We distinguish between a set of explanatory vari- ames. and one response variable. A statistical model can be used to perforin prediction in the following manner: Given the values of the explanatory vari- ables, what is the probability distribution for the response variable, i.e.. what are the probabilities for the different possible values of the response variable? 2.2 The Contingency Table Tile ba,sic tool used in categorical data analysis is the contingency table (sometimes called the "cross- classified table of counts"). A contingency table is a matrix with one dimension for each variable, …

Content truncated to 500 words