A Joint Language Model With Fine-grain Syntactic Tags

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1114–1123, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP A Joint Language Model With Fine-grain Syntactic Tags Denis Filimonov1 1Laboratory for Computational Linguistics and Information Processing Institute for Advanced Computer Studies University of Maryland, College Park den@cs.umd.edu Mary Harper1,2 2Human Language Technology Center of Excellence Johns Hopkins University mharper@umiacs.umd.edu Abstract We present a scalable joint language model designed to utilize fine-grain syn- tactic tags. We discuss challenges such a design faces and describe our solutions that scale well to large tagsets and cor- pora. We advocate the use of relatively simple tags that do not require deep lin- guistic knowledge of the language but pro- vide more structural information than POS tags and can be derived from automati- cally generated parse trees – a combina- tion of properties that allows easy adop- tion of this model for new languages. We propose two fine-grain tagsets and evalu- ate our model using these tags, as well as POS tags and SuperARV tags in a speech recognition task and discuss future direc- tions. 1 Introduction In a number of language processing tasks, particu- larly automatic speech recognition (ASR) and ma- chine translation (MT), there is the problem of se- lecting the best sequence of words from multiple hypotheses. This problem stems from the noisy channel approach to these applications. The noisy channel model states that the observed data, e.g., the acoustic signal, is the result of some input translated by some unknown stochastic process. Then the problem of finding the best sequence of words given the acoustic input, not approachable directly, is transformed into two separate models: argmax wn 1 p(wn 1 |A) = argmax wn 1 p(A|wn 1 ) · p(wn 1 ) (1) where A is the acoustic signal and wn 1 is a se- quence of n words. p(A|wn 1 ) is called an acoustic model and p(wn 1 ) is the language model1. Typically, these applications use language mod- els that compute the probability of a sequence in a generative way: p(wn 1 ) = n � i=1 p(wi|wi−1 1 ) Approximation is required to keep the parameter space tractable. Most commonly the context is re- duced to just a few immediately preceding words. This type of model is called an ngram model: p(wi|wi−1 1 ) ≈ p(wi|wi−1 i−n+1) Even with limited context, the parameter space can be quite sparse and requires sophisticated tech- niques for reliable probability estimation (Chen and Goodman, 1996). While the ngram models perform fairly well, they are only capable of cap- turing very shallow knowledge of the language. There is extensive literature on a variety of methods that have been used to imbue models with syntactic and semantic information in differ- ent ways. These methods can be broadly catego- rized into two types: • The first method uses surface words within its context, sometimes organizing them into deterministic classes. Models of this type in- clude: (Brown et al., 1992; Zitouni, 2007), …

Content truncated to 500 words