A Character-Level Machine Translation Approach for Normalization of SMS Abbreviations

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 974–982, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP A Character-Level Machine Translation Approach for Normalization of SMS Abbreviations Deana L. Pennell The University of Texas at Dallas 800 W. Campbell Road Richardson, TX 75080 deana@hlt.utdallas.edu Yang Liu The University of Texas at Dallas 800 W. Campbell Road Richardson, TX 75080 yangl@hlt.utdallas.edu Abstract This paper describes a two-phase method for expanding abbreviations found in in- formal text (e.g., email, text messages, chat room conversations) using a machine translation system trained at the charac- ter level during the first phase. In this way, the system learns mappings between character-level “phrases” and is much more robust to new abbreviations than a word-level system. We generate transla- tion models that are independent of the way in which the abbreviations are formed and show that the results show little degra- dation compared to when type-dependent models are trained. Our experiments on a large data set show our proposed system performs well when tested both on isolated abbreviations and, with the incorporation of a second phase utilizing an in-domain language model, in the context of neigh- boring words. 1 Introduction Text messaging (SMS) is a rapidly growing form of alternative communication for cell phones. This popularity has caused safety concerns leading many US states to pass laws prohibiting texting while driving. The technology is also difficult for users with visual impairments or physical hand- icaps to use. We believe a text-to-speech (TTS) system for cell phones can decrease these prob- lems to promote safe travel and ease of use for all. Text normalization is the usual first step for TTS. Text message lingo is also similar to the chat- speak that is prolific on forums, blogs and chat- rooms. Screen readers will thus benefit from such technology, enabling visually impaired users to take part in internet culture. In addition, normal- izing informal text is important for various lan- guage processing tasks, such as information re- trieval, summarization, and keyword, topic, sen- timent and emotion detection, which are currently receiving a lot of attention for informal domains. Normalization of informal text is complicated by the large number of abbreviations used. Some previous work on this problem used phrase-based machine translation (MT) for SMS normalization; however, a large annotated corpus is required for such a supervised learning method since the learn- ing is performed at the word level. By definition, this method cannot make a hypothesis for an ab- breviation it did not see in training. This is a se- rious limitation in a domain where new words are created frequently and irregularly. We propose a two-phase approach. In the first phase, an MT model is trained at the character- level rather than the word- or phrase-level, allow- ing recognition of common abbreviation patterns regardless of the words in which they appear. We decode the hypotheses in the second phase using a language model for the final prediction. 2 Related Work Text normalization …

Content truncated to 500 words