N-gram Probabilities

We present an algorithm for computing n-gram probabil- ities from stochastic context-free grammars, a procedure that can alleviate some of the standard problems associated with n-grams (estimation from sparse data, …...

Non-linear mappings of the form P(ngram)γ and log(1+τP(ngram)) log(1+τ) are applied to the n-gram probabilities in five trainable open-source language identifiers. The first mapping reduces classification errors by 4.0% to …...