Learning a syntagmatic and paradigmatic structure from language data with a bi-multigram model

Learning a syntagmatic and paradigmatic structure from language data with a bi-multigram model Sabine Deligne and Yoshinori Sagisaka ATR-ITL, deptl, 2-2 Hikaridai Seika cho, Soraku gun, Kyoto fu 619-0224, Japan. Abstract In this paper, we present a stochastic language mod- eling tool which aims at retrieving variable-length phrases (multigrams), assuming bigram dependen- cies between them. The phrase retrieval can be in- termixed with a phrase clustering procedure, so that the language data are iteratively structured at both a paradigmatic and a syntagmatic level in a fully in- tegrated way. Perplexity results on ATR travel ar- rangement data with a bi-multigram model (assum- ing bigram correlations between the phrases) come very close to the trigram scores with a reduced num- ber of entries in the language model. Also the ability of the class version of the model to merge semanti- cally related phrases into a common class is illus- trated. 1 Introduction There is currently an increasing interest in statisti- cal language models, which in one way or another aim at exploiting word-dependencies spanning over a variable number of words. Though all these mod- els commonly relax the assumption of fixed-length dependency of the conventional ngram model, they cover a wide variety of modeling assumptions and of parameter estimation frameworks. In this paper, we focus on a phrase-based approach, as opposed to a gram-based approach: sentences are structured into phrases and probabilities are assigned to phrases in- stead of words. Regardless of whether they are gram or phrase-based, models can be either determinis- tic or stochastic. In the phrase-based framework, non determinism is introduced via an ambiguity on the parse of the sentence into phrases. In practice, it means that even if phrase abe is registered as a phrase, the possibility of parsing the string as, for instance, [ab] [c] still remains. By contrast, in a de- terministic approach, all eo-occurences of a, b and c would be systematically interpreted as an occurence of phrase [abc]. Various criteria have been proposed to derive phrases in a purely statistical way 1; data likeli- 1i.e. without using grarmnar rules llke in Stochastic Con- text Free Grammars. hood, leaving-one-out likelihood (Ries et al., 1996), mutual information (Suhm and Waibel, 1994), and entropy (Masataki and Sagisaka, 1996). The use of the likelihood criterion in a stochastic framework al- lows EM principled optimization procedures, but it is prone to over]earning. The other criteria tend to reduce the risk of overlearning, but their opti- mization relies on heuristic procedures (e.g. word grouping via a greedy algorithm (Matsunaga and Sagayama, 1997)) for which convergence and opti- mality are not theoretically guaranteed. The work reported in this paper is based on the multigram model, which is a stochastic phrase-based model, the parameters of which are estimated according to a likelihood criterion using an EM procedure. The multigram approach was introduced in (Bimbot et al., 1995), and in (Deligne and Bimbot, 1995) it was used to derive variable-length phrases under the as- sumption of independence of the phrases. Various ways …

Content truncated to 500 words