Improvement of a Whole Sentence Maximum Entropy Language Model Using Grammatical Features

Improvement of a Whole Sentence Maximum Entropy Language Model Using Grammatical Features � Fredy Amaya � and Jos´e Miguel Bened´ı Departamento de Sistemas Inform´aticos y Computaci´on Universidad Polit´ecnica de Valencia Camino de vera s/n, 46022-Valencia (Spain) � famaya, jbenedi � @dsic.upv.es Abstract In this paper, we propose adding long-term grammatical information in a Whole Sentence Maximun Entropy Language Model (WSME) in order to improve the performance of the model. The grammatical information was added to the WSME model as fea- tures and were obtained from a Stochas- tic Context-Free grammar. Finally, ex- periments using a part of the Penn Tree- bank corpus were carried out and sig- nificant improvements were acheived. 1 Introduction Language modeling is an important component in computational applications such as speech recog- nition, automatic translation, optical character recognition, information retrieval etc. (Jelinek, 1997; Borthwick, 1997). Statistical language models have gained considerable acceptance due to the efficiency demonstrated in the fields in which they have been applied (Bahal et al., 1983; Jelinek et al., 1991; Ratnapharkhi, 1998; Borth- wick, 1999). Traditional statistical language models calcu- late the probability of a sentence � using the chain rule: �� �� ��    �      ��   � (1) This work has been partially supported by the Spanish CYCIT under contract (TIC98/0423-C06). ! Granted by Universidad del Cauca, Popay´an (Colom- bia) where " #  $  , which is usually known as the history of  . The effort in the language modeling techniques is usually directed to the es- timation of ��  " � . The language model defined by the expression ��  " � is named the condi- tional language model. In principle, the deter- mination of the conditional probability in (1) is expensive, because the possible number of word sequences is very great. Traditional conditional language models assume that the probability of the word  does not depend on the entire history, and the history is limited by an equivalence rela- tion % , and (1) is rewritten as: �� ��& ��     �&'   (  ��  % �  �� (2) The most commonly used conditional language model is the n-gram model. In the n-gram model, the history is reduced (by the equivalence rela- tion) to the last )+*-, words. The power of the n-gram model resides in: its consistence with the training data, its simple formulation, and its easy implementation. However, the n-gram model only uses the information provided by the last ).*/, words to predict the next word and so only makes use of local information. In addition, the value of n must be low ( 021 ) because for )43-1 there are problems with the parameter estimation. Hybrid models have been proposed, in an at- tempt to supplement the local information with long-distance information. They combine dif- ferent types of models, like n-grams, with long- distance information, generally by means of lin- ear interpolation, as has been …

Content truncated to 500 words