A Hybrid Rule/Model-Based Finite-State Framework for Normalizing SMS Messages

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 770–779, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics A hybrid rule/model-based finite-state framework for normalizing SMS messages Richard Beaufort1 Sophie Roekhaut2 Louise-Amélie Cougnon1 Cédrick Fairon1 (1) CENTAL, Université catholique de Louvain – 1348 Louvain-la-Neuve, Belgium {richard.beaufort,louise-amelie.cougnon,cedrick.fairon}@uclouvain.be (2) TCTS Lab, Université de Mons – 7000 Mons, Belgium sophie.roekhaut@umons.ac.be Abstract In recent years, research in natural language processing has increasingly focused on normalizing SMS messages. Different well-defined approaches have been proposed, but the problem remains far from being solved: best systems achieve a 11% Word Error Rate. This paper presents a method that shares similarities with both spell checking and machine translation approaches. The normalization part of the system is entirely based on models trained from a corpus. Evaluated in French by 10-fold-cross validation, the system achieves a 9.3% Word Error Rate and a 0.83 BLEU score. 1 Introduction Introduced a few years ago, Short Message Service (SMS) offers the possibility of exchanging written messages between mobile phones. SMS has quickly been adopted by users. These messages often greatly deviate from traditional spelling conventions. (Thurlow and Brown,As shown by specialists 2003; Fairon et al., 2006; Bieswanger, 2007), this variability is due to the simultaneous use of numerous coding strategies, like phonetic plays (2m1 read ‘demain’, “tomorrow”), phonetic transcriptions (kom instead of ‘comme’, “like”), consonant skeletons (tjrs for ‘toujours’, “always”), misapplied, missing or incorrect separators (j esper for ‘j’espère’, “I hope”; j’croibi1k, instead of ‘je crois bien que’, “I am pretty sure that”), etc. These deviations are due to three main factors: the small number of characters allowed per text message by the service (140 bytes), the constraints of the small phones’ keypads and, last but not least, the fact that people mostly communicate between friends and relatives in an informal register. Whatever their causes, these deviations considerably hamper any standard natural language processing (NLP) system, which stumbles against so many Out-Of-Vocabulary words. For this reason, as noted by Sproat et al. (2001), an SMS normalization must be performed before a more conventional NLP process can be applied. As defined by Yvon (2008), “SMS normalization consists in rewriting an SMS text using a more conventional spelling, in order to make it more readable for a human or for a machine.” The SMS normalization we present here was developed in the general framework of an SMS- to-speech synthesis system1. This paper, however, only focuses on the normalization process. Evaluated in French, our method shares similarities with both spell checking and machine translation. The machine translation-like module of the system performs the true normalization task. It is entirely based on models learned from an SMS corpus and its transcription, aligned at the character-level in order to get parallel corpora. Two spell checking-like modules surround the normalization module. The first one detects unambiguous tokens, like URLs or phone numbers, to keep them out of the normalization. The second one, applied on the normalized parts only, identifies non-alphabetic sequences, like punctuations, …

Content truncated to 500 words