Non-linear Mapping for Improved Identification of 1300+ Languages

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 627–632, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Non-linear Mapping for Improved Identification of 1300+ Languages Ralf D. Brown Carnegie Mellon University Language Technologies Institute 5000 Forbes Avenue, Pittsburgh PA 15213 USA ralf@cs.cmu.edu Abstract Non-linear mappings of the form P(ngram)γ and log(1+τP(ngram)) log(1+τ) are applied to the n-gram probabilities in five trainable open-source language identifiers. The first mapping reduces classification errors by 4.0% to 83.9% over a test set of more than one million 65-character strings in 1366 languages, and by 2.6% to 76.7% over a subset of 781 languages. The second mapping improves four of the five identifiers by 10.6% to 83.8% on the larger corpus and 14.4% to 76.7% on the smaller corpus. The subset corpus and the modified programs are made freely available for download at http://www.cs.cmu.edu/∼ralf/langid.html. 1 Introduction Language identification, particularly of short strings, is a task which is becoming quite impor- tant as a preliminary step in much automated pro- cessing of online data streams such as microblogs (e.g. Twitter). In addition, an increasing num- ber of languages are represented online, so it is desireable that performance remain high as more languages are added to the identifier. In this paper, we stress-test five open-source n-gram-based language identifiers by presenting them with 65-character strings (about one printed line of text in a book) in up to 1366 languages. We then apply a simple modification to their scoring algorithms which improves the classification ac- curacy of all five of them, three quite dramatically. 2 Method The selected modification to the scoring algorithm is to apply a non-linear mapping which spreads out the lower probability values while compact- ing the higher ones. This low-end spreading of values is the opposite of what one sees in a Zip- fian distribution (Zipf, 1935), where the proba- bilities of the most common items are the most spread out while the less frequent items become ever more crowded as there are increasing num- bers of them in ever-smaller ranges. The hypoth- esis is that regularizing the spacing between val- ues will improve language-identification accuracy by avoiding over-weighting frequent items (from having higher probabilities in the training data and also occurring more frequently in the test string). Two functions were selected for experiments: x = P(ngram) gamma: y = xγ loglike: y = log(1 + 10τx) log(1 + 10τ) The first simply raises the n-gram probabil- ity to a non-unity power; this exponent is named “gamma” as in image processing (Poynton, 1998). The second mapping function is a normalized vari- ant of the logarithm function; the normalization provides fixed points at 0 and 1, as is the case for gamma. Each of the functions gamma and loglike has one tunable parameter, γ and τ, respectively. 3 Related Work Although n-gram statistics as a basis for language identification has been in use for two decades since Cavnar and Trenkle (1994) and Dunning (1994), little work has been …

Content truncated to 500 words