A Log-Linear Model for Unsupervised Text Normalization

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 61–72, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics A Log-Linear Model for Unsupervised Text Normalization Yi Yang School of Interactive Computing Georgia Institute of Technology yiyang@gatech.edu Jacob Eisenstein School of Interactive Computing Georgia Institute of Technology jacobe@gatech.edu Abstract We present a unified unsupervised statistical model for text normalization. The relation- ship between standard and non-standard to- kens is characterized by a log-linear model, permitting arbitrary features. The weights of these features are trained in a maximum- likelihood framework, employing a novel se- quential Monte Carlo training algorithm to overcome the large label space, which would be impractical for traditional dynamic pro- gramming solutions. This model is im- plemented in a normalization system called UNLOL, which achieves the best known re- sults on two normalization datasets, outper- forming more complex systems. We use the output of UNLOL to automatically normalize a large corpus of social media text, revealing a set of coherent orthographic styles that under- lie online language variation. 1 Introduction Social media language can differ substantially from other written text. Many of the attempts to character- ize and overcome this variation have focused on nor- malization: transforming social media language into text that better matches standard datasets (Sproat et al., 2001; Liu et al., 2011). Because there is lit- tle available training data, and because social me- dia language changes rapidly (Eisenstein, 2013b), fully supervised training is generally not considered appropriate for this task. However, due to the ex- tremely high-dimensional output space — arbitrary sequences of words across the vocabulary — it is a very challenging problem for unsupervised learn- ing. Perhaps it is for these reasons that the most suc- cessful systems are pipeline architectures that cob- ble together a diverse array of techniques and re- sources, including statistical language models, de- pendency parsers, string edit distances, off-the-shelf spellcheckers, and curated slang dictionaries (Liu et al., 2011; Han and Baldwin, 2011; Han et al., 2013). We propose a different approach, performing nor- malization in a maximum-likelihood framework. There are two main sources of information to be exploited: local context, and surface similarity be- tween the observed strings and normalization can- didates. We treat the local context using standard language modeling techniques; we treat string simi- larity with a log-linear model that includes features for both surface similarity and word-word pairs. Because labeled examples of normalized text are not available, this model cannot be trained in the standard supervised fashion. Nor can we apply dynamic programming techniques for unsu- pervised training of locally-normalized conditional models (Berg-Kirkpatrick et al., 2010), as their com- plexity is quadratic in the size of label space; in normalization, the label space is the vocabulary it- self, with at least 104 elements. Instead, we present a new training approach using Monte Carlo tech- niques to compute an approximate gradient on the feature weights. This training method may be appli- cable in other unsupervised learning problems with a …

Content truncated to 500 words