Beyond Normalization: Pragmatics of Word Form in Text Messages

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1437–1441, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Beyond Normalization: Pragmatics of Word Form in Text Messages Tyler Baldwin Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824 baldwi96@cse.msu.edu Joyce Y. Chai Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824 jchai@cse.msu.edu Abstract Non-standard spellings in text messages often convey extra pragmatic information not found in the standard word form. However, text message normalization sys- tems that transform non-standard text mes- sage spellings to standard form tend to ignore this information. To address this problem, this paper examines the types of extra pragmatic information that are con- veyed by non-standard word forms. Em- pirical analysis of our data shows that 40% of non-standard word forms contain emo- tional information not found in the stan- dard form, and 38% contain additional emphasis. This extra information can be important to downstream applications such as text-to-speech synthesis. We fur- ther investigated the automatic detection of non-standard forms that display addi- tional information. Our empirical results show that character level features can pro- vide important cues for such detection. 1 Introduction Text message conversations are often filled with non-standard word spellings. While some of these are unintentional misspellings, many of them are purposely produced. One commonly acknowl- edged reason that text message authors intention- ally use non-standard word forms is to reduce the amount of time it takes to type the message, or the amount of space the message occupies. This phenomenon has motivated the text mes- sage normalization task (Aw et al., 2006), which attempts to replace non-standard spelling and symbols by their standard forms. The normaliza- tion task is potentially critical for applications in- volving text messages, such as text-to-speech syn- thesis. However, one important aspect that is over- looked when performing normalization is the use of non-standard word forms to express additional information such as emotion or emphasis. For in- stance, consider the following text message con- versation: A: They won the game! B: Yesssss The intent of the utterance by person B seems clear: he wishes to show that he is happy about the event described by person A. If the non-standard form Yesssss was normalized to the standard form yes, the intent conveyed by the utterance would be ambiguous; it could suggest that person B is happy about this turn of events, or he is indifferent, or he could simply be acknowledging that he already knows this fact. By using the non-standard form instead of the standard one, Person B communi- cated his excitement to A. As shown in the above example, text message users often employ these non-standard forms to display extra pragmatic information that is not easily displayed otherwise. However, because normalization is only concerned about converting non-standard spellings to standard forms, it has the potential to remove this important pragmatic infor- mation. To address this problem and to better under- stand some of …

Content truncated to 500 words