Automatic Acquisition of Language Model based on Head-Dependent Relation between Words

Automatic Acquisition of Language Model based on Head-Dependent Relation between Words Seungmi Lee ~nd Key-Sun Choi Department of Computer Science Center for Artificial Intelligence Research Korea Advanced Institute of Science a.nd Technology e-mail: {leesm, kschoi}©world.kaist.ac.kr Abstract Language modeling is to associate a sequence of words with a priori probability, which is a key part of many natural language applications such as speech recognition and statisticM lna- chine translation. In this paper, we present a language modeling based on a kind of simple dependency grammar. Tile grammar consists of hea.d-dependellt relations between words and cau be learned automatically from a raw corpus using the reestimation algorithm which is also introduced in this paper. Our experiments show that the proposed model performs better than u-gram models at 11% to 11.5% reductions in test corpus entropy. 1 Introduction Language modeling is to associate a priori prob- ttbility to a sentence. It is a key part of many natural language applications such a,s speech recognition a.nd sta, tistica, l machine translation. Previous works for lallguage modeling Call be broadly divided into two approaches; oue is 11- gram-based and the other is grammar-I)ased. N-gram model estinlates the probability of a sentence a.s the product of tile probability of each word in tile sentence. It, assumes that probM)ility of the nth word is dependent on the previous n - 1 words. The n-gram prob- abilities are estimated by simply counting the n-gram frequencies in a trMnillg corpus. In some cases, class (or part of speech) 11-grams are used instead of word n-grams(Brown et al., 1992; Chang and Cheu, 1996). N-gram model has beeu widely used so far, but; it has always been clear that n-gram can not represent long distance dependencies. In contrast with n-gram model, grammar- based approa.ch assigns syntactic structures to a sentence and computes the l)robability of tile sentence using the probabilities of the struc- tures. Lollg distance dependencies can be rep- resented well by means of the structures. Tile approach usuMly makes use of phrase struc- ture grammars such as probabilistic context-free grammar and recursive transition 11etwork(Lari a11d Young, 1.991; Snefl', 1992; Chen, 1996). In the approach, however, a sentence which is not accepted by the grammar is assigned zero prob- M)ility. Thus, the grammar nlust have broad- coverage so that any sentence will get non-zero probability. But acquisition of such a. robust grammar has been ktlown to be very difficult. Due to tile difficulty, some works try to use an integrated model of grammar and l>gram com- I)ensa.tillg each other(McCandless, 1994; Meteer and R.ohlicek, 1993). Given a robust grammar, gramnmr-based language nmdeling is expected to be more powerful and compact in model size than n-gram-based one. In this paper we present a lallguage modeling based on a kind of simple dependency gram- ma.r. The grammar consists of head-delmndent relatiolls between words and ca11 be learned au- tomatically from a ra.w corpus using the rees- tima.tion algorithm which is Mso introduced in this paper. Based on the dependencies, a sen- tellce is …

Content truncated to 500 words