Linguistic Theory in Statistical Language Learning

II i Hi HI m i m i m ill IH i IH IH HI I I I I i I Linguistic Theory in Statistical Language Learning Christer Samuelsson Bell Laboratories, Lucent Technologies 600 Mountain Ave, Room 2D-339, Murray Hill, NJ 07974, USA christer@research, bell-labs, tom Abstract This article attempts to determine what ele- ments of linguistic theory are used in statisti- cal language learning, and why the extracted language models look like they do. The study indicates that some linguistic elements, such as the notion of a word, are simply too useful to be ignored. The second most important factor seems to be features inherited from the origi- nal task for which the technique was used, for example using hidden Markov models for part- of-speech tagging, rather than speech recogni- tion. The two remaining important factors are properties of the runtime processing scheme employing the extracted language model, and the properties of the available corpus resources to which the statistical learning techniques are applied. Deliberate attempts to include lin- guistic theory seem to end up in a fifth place. 1 Introduction What role does linguistics play in statistical lan- guage learning? "None at all!" might be the answer, if we ask hard-core speech-recognition professionals. But even the most nonlinguistic language model, for example a statistic word bigram model, actually re- lies on key concepts integral to virtually all linguistic theories. Words, for example, and the notion that sequences of words form utterances. Statistical language learning is applied to some set of data to extract a language model of some kind. This language model can serve a purely decorative purpose, but is more often than not used to pro- cess data in some way, for example to aid speech recognition. Anyone working under the pressure of producing better results, and who employs language models to this purpose, such a researchers in the field of speech recognition, will have a high incen- tive of incorporating useful aspects of language into his or her language models. Now, the most useful, and thus least controversial ways of describing lan- guage will, due to their usefulness, find their way into most linguistic theories and, for the very same reason, be used in models that strive to model lan- guage successfully. So what do the linguistic theories underlying vari- ous statistical language models look like? And why? It may be useful to distinguish between those aspects of linguistic theory that are incidentally in the lan- guage model, and those that are there intentionally. We will start our tour of statistical language learn- ing by inspecting language models with "very little" linguistic content, and then proceed to analyse in- creasingly more linguistic models, until we end with models that are entirely linguistic, in the sense that they are pure grammars, associated with no statis- tical parameters. 2 Word N-gr_~m Models Let us return to the simple bigram word model, where the probability of each next word is deter- mined from the current one. We already noted …

Content truncated to 500 words