Lexical Normalisation of Short Text Messages: Makn Sens a #twitter

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 368–378, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Lexical Normalisation of Short Text Messages: Makn Sens a #twitter Bo Han and Timothy Baldwin NICTA Victoria Research Laboratory Department of Computer Science and Software Engineering The University of Melbourne hanb@student.unimelb.edu.au tb@ldwin.net Abstract Twitter provides access to large volumes of data in real time, but is notoriously noisy, hampering its utility for NLP. In this paper, we target out-of-vocabulary words in short text messages and propose a method for identify- ing and normalising ill-formed words. Our method uses a classifier to detect ill-formed words, and generates correction candidates based on morphophonemic similarity. Both word similarity and context are then exploited to select the most probable correction can- didate for the word. The proposed method doesn’t require any annotations, and achieves state-of-the-art performance over an SMS cor- pus and a novel dataset based on Twitter. 1 Introduction Twitter and other micro-blogging services are highly attractive for information extraction and text mining purposes, as they offer large volumes of real-time data, with around 65 millions tweets posted on Twit- ter per day in June 2010 (Twitter, 2010). The quality of messages varies significantly, however, ranging from high quality newswire-like text to meaningless strings. Typos, ad hoc abbreviations, phonetic sub- stitutions, ungrammatical structures and emoticons abound in short text messages, causing grief for text processing tools (Sproat et al., 2001; Ritter et al., 2010). For instance, presented with the input u must be talkin bout the paper but I was thinkin movies (“You must be talking about the paper but I was thinking movies”),1 the Stanford parser (Klein and 1Throughout the paper, we will provide a normalised version of examples as a gloss in double quotes. Manning, 2003; de Marneffe et al., 2006) analyses bout the paper and thinkin movies as a clause and noun phrase, respectively, rather than a prepositional phrase and verb phrase. If there were some way of preprocessing the message to produce a more canon- ical lexical rendering, we would expect the quality of the parser to improve appreciably. Our aim in this paper is this task of lexical normalisation of noisy English text, with a particular focus on Twitter and SMS messages. In this paper, we will collectively refer to individual instances of typos, ad hoc abbre- viations, unconventional spellings, phonetic substi- tutions and other causes of lexical deviation as “ill- formed words”. The message normalisation task is challenging. It has similarities with spell checking (Peterson, 1980), but differs in that ill-formedness in text mes- sages is often intentional, whether due to the desire to save characters/keystrokes, for social identity, or due to convention in this text sub-genre. We propose to go beyond spell checkers, in performing deabbre- viation when appropriate, and recovering the canon- ical word form of commonplace shorthands like b4 “before”, which tend to be considered beyond the remit of spell checking (Aw et al., 2006). The free writing style of …

Content truncated to 500 words