Social Text Normalization using Contextual Graph Random Walks

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1577–1586, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Social Text Normalization using Contextual Graph Random Walks Hany Hassan Microsoft Research Redmond, WA hanyh@microsoft.com Arul Menezes Microsoft Research Redmond, WA arulm@microsoft.com Abstract We introduce a social media text normal- ization system that can be deployed as a preprocessing step for Machine Transla- tion and various NLP applications to han- dle social media text. The proposed sys- tem is based on unsupervised learning of the normalization equivalences from unla- beled text. The proposed approach uses Random Walks on a contextual similarity bipartite graph constructed from n-gram sequences on large unlabeled text corpus. We show that the proposed approach has a very high precision of (92.43) and a rea- sonable recall of (56.4). When used as a preprocessing step for a state-of-the-art machine translation system, the translation quality on social media text improved by 6%. The proposed approach is domain and language independent and can be deployed as a preprocessing step for any NLP appli- cation to handle social media text. 1 Introduction Social Media text is usually very noisy and con- tains a lot of typos, ad-hoc abbreviations, pho- netic substitutions, customized abbreviations and slang language. The social media text is evolving with new entities, words and expressions. Natural language processing and understanding systems such as Machine Translation, Information Extrac- tion and Text-to-Speech are usually trained and optimized for clean data; therefore such systems would face a challenging problem with social me- dia text. Various social media genres developed distinct characteristics. For example, SMS developed a nature of shortening messages to avoid multiple keystrokes. On the other hand, Facebook and in- stant messaging developed another genre where more emotional expressions and different abbre- viations are very common. Somewhere in be- tween, Twitter’s statuses come with some brevity similar to SMS along with the social aspect of Facebook. On the same time, various social me- dia genres share many characteristics and typo styles. For example, repeating letters or punctu- ation for emphasizing and emotional expression such as ”‘goooood morniiing”’. Using phonetic spelling in a generalized way or to reflect a lo- cal accent; such as ”‘wuz up bro”’ (what is up brother). Eliminating vowels such as ”‘cm to c my luv”’. Substituting numbers for letters such as ”‘4get”’ (forget) , ”‘2morrow”’ (tomorrow), and ”‘b4”’ (before). Substituting phonetically sim- ilar letters such as ”‘phone”’ (fon). Slang ab- breviations which usually abbreviates multi-word expression such as ”‘LMS”’ (like my status) , ”‘idk”’ (i do not know), ”‘rofl”’ (rolling on floor laughing). While social media genres share many charac- teristics, they have significant differences as well. It is crucial to have a solution for text normaliza- tion that can adapt to such variations automati- cally. We propose a text normalization approach using an unsupervised method to induce normal- ization equivalences from noisy data which can adapt to any genre of social media. In this paper, we focus on providing a …

Content truncated to 500 words