A Framework for Translating SMS Messages

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 974–983, Dublin, Ireland, August 23-29 2014. A Framework for Translating SMS Messages Vivek Kumar Rangarajan Sridhar, John Chen, Srinivas Bangalore, Ron Shacham AT&T Labs 1 AT&T Way, Bedminster, NJ 07921 vkumar,jchen,srini,rshacham@research.att.com Abstract Short Messaging Service (SMS) has become a popular form of communication. While it is predominantly used for monolingual communication, it can be extremely useful for facilitating cross-lingual communication through statistical machine translation. In this work we present an application of statistical machine translation to SMS messages. We decouple the SMS transla- tion task into normalization followed by translation so that one can exploit existing bitext re- sources and present a novel unsupervised normalization approach using distributed representa- tion of words learned through neural networks. We describe several surrogate data that are good approximations to real SMS data feeds and use a hybrid translation approach using finite-state transducers. Both objective and subjective evaluation indicate that our approach is highly suitable for translating SMS messages. 1 Introduction The preferred form of communication has been changing over time with advances in communication technology. The majority of the world’s population now owns a mobile device and an ever increasing fraction of users are resorting to Short Message Service (SMS) as the primary form of communication. SMS offers an easy, convenient and condensed form of communication that is being embraced by the younger demographic. Due to the inherent limit in the length of a message that can be transmitted, SMS users have adopted several shorthand notations to compress the message; some that have become standardized and many that are invented constantly. While SMS is predominantly used in a monolingual mode, it has the potential to connect people speaking different languages. However, translating SMS messages has several challenges ranging from the procurement of data in this domain to dealing with noisy text (abbreviations, spelling errors, lack of punctuation, etc.) that is typically detrimental to trans- lation quality. In this work we address all the elements involved in building a cross-lingual SMS service that spans data acquisition, normalization, translation modeling, messaging infrastructure and user trial. The rest of the paper is organized as follows. In Section 4, we present a variety of channels through which we compiled SMS data followed by a description of our pipeline in Section 5 that includes nor- malization, phrase segmentation and machine translation. Finally, we describe a SMS translation service built using our pipeline in Section 6 along with results from a user trial. We provide some discussion in Section 7 and conclude in Section 8. 2 Related Work One of the main challenges of building a machine translation system for SMS messages is the lack of training data in this domain. Typically, there are several legal restrictions in using consumer SMS data that precludes one from either using it completely or forces one to use it in limited capacity. Only a handful of such corpora are publicly available on the Web (Chen and Kan, 2013; …

Content truncated to 500 words