Automatic Learning of Language Model Structure

Automatic Learning of Language Model Structure Kevin Duh and Katrin Kirchhoff Department of Electrical Engineering University of Washington, Seattle, USA {duh,katrin}@ee.washington.edu Abstract Statistical language modeling remains a challeng- ing task, in particular for morphologically rich lan- guages. Recently, new approaches based on factored language models have been developed to address this problem. These models provide principled ways of including additional conditioning variables other than the preceding words, such as morphological or syntactic features. However, the number of possible choices for model parameters creates a large space of models that cannot be searched exhaustively. This paper presents an entirely data-driven model selec- tion procedure based on genetic search, which is shown to outperform both knowledge-based and ran- dom selection procedures on two different language modeling tasks (Arabic and Turkish). 1 Introduction In spite of novel algorithmic developments and the increased availability of large text corpora, statistical language modeling remains a diffi- cult problem, particularly for languages with rich morphology. Such languages typically ex- hibit a large number of word types in relation to word tokens in a given text, which leads to high perplexity and a large number of un- seen word contexts. As a result, probability es- timates are often unreliable, even when using standard smoothing and parameter reduction techniques. Recently, a new language model- ing approach, called factored language models (FLMs), has been developed. FLMs are a gen- eralization of standard language models in that they allow a larger set of conditioning variables for predicting the current word. In addition to the preceding words, any number of additional variables can be included (e.g. morphological, syntactic, or semantic word features). Since such features are typically shared across mul- tiple words, they can be used to obtained bet- ter smoothed probability estimates when train- ing data is sparse. However, the space of pos- sible models is extremely large, due to many different ways of choosing subsets of condition- ing word features, backoff procedures, and dis- counting methods. Usually, this space cannot be searched exhaustively, and optimizing mod- els by a knowledge-inspired manual search pro- cedure often leads to suboptimal results since only a small portion of the search space can be explored. In this paper we investigate the possibility of determining the structure of fac- tored language models (i.e. the set of condition- ing variables, the backoff procedure and the dis- counting parameters) by a data-driven search procedure, viz. Genetic Algorithms (GAs). We apply this technique to two different tasks (lan- guage modeling for Arabic and Turkish) and show that GAs lead to better models than ei- ther knowledge-inspired manual search or ran- dom search. The remainder of this paper is structured as follows: Section 2 describes the details of the factored language modeling ap- proach. The application of GAs to the problem of determining language model structure is ex- plained in Section 3. The corpora used in the present study are described in Section 4 and ex- periments and results are presented in Section 5. Section 6 compares the present …

Content truncated to 500 words