Automatic Acquisition of Language Model based on Head-Dependent Relation between Words

Automatic Acquisition of Language Model based on Head-Dependent Relation between Words Seungmi Lee and Key-Sun Choi Department of Computer Science Center for Artificial Intelligence Research Korea Advanced Institute of Science and Technology e-mail: {leesm, kschoi}@world, kaist, ac. kr Abstract Language modeling is to associate a sequence of words with a priori probability, which is a key part of many natural language applications such as speech recognition and statistical ma- chine translation. In this paper, we present a language modeling based on a kind of simple dependency grammar. The grammar consists of head-dependent relations between words and can be learned automatically from a raw corpus using the reestimation algorithm which is also introduced in this paper. Our experiments show that the proposed model performs better than n-gram models at 11% to 11.5~ reductions in test corpus entropy. 1 Introduction Language modeling is to associate a priori prob- ability to a sentence. It is a key part of many natural language applications such as speech recognition and statistical machine translation. Previous works for language modeling can be broadly divided into two approaches; one is n- gram-based and the other is grammar-based. N-gram model estimates the probability of a sentence as the product of the probability of each word in the sentence. It assumes that probability of the nth word is dependent on the previous n- 1 words. The n-gram prob- abilities are estimated by simply counting the n-gram frequencies in a training corpus. In some cases, class (or part of speech) n-grams are used instead of word n-grams(Brown et al., 1992; Chang and Chen, 1996). N-gram model has been widely used so far, but it has always been clear that n-gram can not represent long distance dependencies. In contrast with n-gram model, grammar- based approach assigns syntactic structures to a sentence and computes the probability of the sentence using the probabilities of the struc- tures. Long distance dependencies can be rep- resented well by means of the structures. The approach usually makes use of phrase struc- ture grammars such as probabilistic context-free grammar and recursive transition network(Lari and Young, 1991; Sneff, 1992; Chen, 1996). In the approach, however, a sentence which is not accepted by the grammar is assigned zero prob- ability. Thus, the grammar must have broad- coverage so that any sentence will get non-zero probability. But acquisition of such a robust grammar has been known to be very difficult. Due to the difficulty, some works try to use an integrated model of grammar and n-gram com- pensating each other(McCandless, 1994; Meteer and Rohlicek, 1993). Given a robust grammar, grammar-based language modeling is expected to be more powerful and compact in model size than n-gram-based one. In this paper we present a language modeling based on a kind of simple dependency gram- mar. The grammar consists of head-dependent relations between words and can be learned au- tomatically from a raw corpus using the rees- timation algorithm which is also introduced in this paper. Based on the dependencies, a sen- tence is …

Content truncated to 500 words