A Graph-based Approach for Contextual Text Normalization

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 313–324, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics A Graph-based Approach for Contextual Text Normalization C¸ a˘gıl S¨onmez and Arzucan ¨Ozg¨ur Department of Computer Engineering Bogazici University Bebek, 34342 Istanbul, Turkey {cagil.ulusahin,arzucan.ozgur}@boun.edu.tr Abstract The informal nature of social media text renders it very difficult to be automati- cally processed by natural language pro- cessing tools. Text normalization, which corresponds to restoring the non-standard words to their canonical forms, provides a solution to this challenge. We introduce an unsupervised text normalization approach that utilizes not only lexical, but also con- textual and grammatical features of social text. The contextual and grammatical fea- tures are extracted from a word association graph built by using a large unlabeled so- cial media text corpus. The graph encodes the relative positions of the words with re- spect to each other, as well as their part-of- speech tags. The lexical features are ob- tained by using the longest common sub- sequence ratio and edit distance measures to encode the surface similarity among words, and the double metaphone algo- rithm to represent the phonetic similarity. Unlike most of the recent approaches that are based on generating normalization dic- tionaries, the proposed approach performs normalization by considering the context of the non-standard words in the input text. Our results show that it achieves state-of- the-art F-score performance on standard datasets. In addition, the system can be tuned to achieve very high precision with- out sacrificing much from recall. 1 Introduction Social text, which has been growing and evolving steadily, has its own lexical and grammatical fea- tures (Choudhury et al., 2007; Eisenstein, 2013). lol meaning laughing out loud, xoxo meaning kiss- ing, 4u meaning for you are among the most com- monly used examples of this jargon. In addition, these informal expressions in social text usually take many different lexical forms when generated by different individuals (Eisenstein, 2013). The limited accuracies of the Speech-to-Text (STT) tools in mobile devices, which are increasingly be- ing used to post messages on social media plat- forms, along with the scarcity of attention of the users result in additional divergence of so- cial text from more standard text such as from the newswire domain. Tools such as spellchecker and slang dictionaries have been shown to be in- sufficient to cope with this challenge long time ago (Sproat et al., 2001). In addition, most Nat- ural Language Processing (NLP) tools including named entity recognizers and dependency parsers generally perform poorly on social text (Ritter et al., 2010). Text normalization is a preprocessing step to restore non-standard words in text to their origi- nal (canonical) forms to make use in NLP applica- tions or more broadly to understand the digitized text better (Han and Baldwin, 2011). For exam- ple, talk 2 u later can be normalized as talk to you later or similarly enormoooos, enrmss and enour- mos can be normalized as enormous. Other exam- ples of …

Content truncated to 500 words