Normalizing SMS: are Two Metaphors Better than One ?

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 441–448 Manchester, August 2008 Normalizing SMS: are two metaphors better than one ? Catherine Kobus Orange Labs 2, avenue Pierre Marzin F-22300 Lannion François Yvon Univ Paris-Sud 11 & LIMSI/CNRS BP 133 F-91403 Orsay Cedex {catherine.kobus,geraldine.damnati}@orange-ftgroup.com, yvon@limsi.fr Géraldine Damnati Orange Labs 2, avenue Pierre Marzin F-22300 Lannion Abstract Electronic written texts used in computer- mediated interactions (e-mails, blogs, chats, etc) present major deviations from the norm of the language. This paper presents an comparative study of systems aiming at normalizing the orthography of French SMS messages: after discussing the linguistic peculiarities of these mes- sages, and possible approaches to their au- tomatic normalization, we present, evalu- ate and contrast two systems, one draw- ing inspiration from the Machine Transla- tion task; the other using techniques that are commonly used in automatic speech recognition devices. Combining both ap- proaches, our best normalization system achieves about 11% Word Error Rate on a test set of about 3000 unseen messages. 1 Introduction The rapid dissemination of electronic communi- cation devices (e-mails, Short Messaging Systems (SMS), chatrooms, instant messaging programs, blogs, etc) has triggered the emergence of new forms of written texts (see eg. (Crystal, 2001; Véronis and Guimier de Neef, 2006)). Addressed to relatives or peers, written on the spur of the moment, using interfaces, each with its specific constraints (computer keyboards, PDAs, mobile phones keypads), these electronic messages are characterised by massive and systematic devia- tions from the orthographic norm, as well as by a non conventional use of alphabetical symbols. c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. In fact, letter and punctuation marks are not only used to conventionally encode a phonetic content, but also to introduce meta-discourse, and to sig- nal emotions, verbal effects (eg. laughters), or at- titudes (humor, derision, emphasis etc). If each media enforces its own set of constraints and pro- motes idiosyncratic forms of writings, these new types of texts nonetheless share a lot of common- alities. To effectively process these messages, it is thus necessary to develop robust language pro- cessing tools, capable of bearing with the extreme form of “noise” they contain. In this study, we focus more specifically on SMS, which, due to the paucity of their input interface (mobile phone keypads) seem to constitute the most challenging type of data. To a large extent, the techniques we present in this paper are also applicable to other types of electronic messages. The “SMS language” (or “texting language”) has been the subject of several linguistic studies (notably, for French, (Anis, 2001; Fairon et al., 2006)), which have emphasized its main charac- teristics, amongst which the extraordinary ortho- graphic variability of lexical forms. In brief, this variability results partly from the mixing of several encoding systems: in SMS, the usual alphabetic system competes with a more “phonetic” type of writing (e.g. rite for right1), as well as with traces of a “consonantic” spelling …

Content truncated to 500 words