Microblogs as Parallel Corpora

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 176–186, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Microblogs as Parallel Corpora Wang Ling123 Guang Xiang2 Chris Dyer2 Alan Black2 Isabel Trancoso 13 (1)L2F Spoken Systems Lab, INESC-ID, Lisbon, Portugal (2)Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA (3)Instituto Superior T´ecnico, Lisbon, Portugal {lingwang,guangx,cdyer,awb}@cs.cmu.edu isabel.trancoso@inesc-id.pt Abstract In the ever-expanding sea of microblog data, there is a surprising amount of naturally occurring par- allel text: some users create post multilingual mes- sages targeting international audiences while oth- ers “retweet” translations. We present an efficient method for detecting these messages and extract- ing parallel segments from them. We have been able to extract over 1M Chinese-English parallel segments from Sina Weibo (the Chinese counter- part of Twitter) using only their public APIs. As a supplement to existing parallel training data, our automatically extracted parallel data yields sub- stantial translation quality improvements in trans- lating microblog text and modest improvements in translating edited news commentary. The re- sources in described in this paper are available at http://www.cs.cmu.edu/∼lingwang/utopia. 1 Introduction Microblogs such as Twitter and Facebook have gained tremendous popularity in the past 10 years. In addition to being an important form of commu- nication for many people, they often contain ex- tremely current, even breaking, information about world events. However, the writing style of mi- croblogs tends to be quite colloquial, with fre- quent orthographic innovation (R U still with me or what?) and nonstandard abbreviations (idk! shm)—quite unlike the style found in more tra- ditional, edited genres. This poses considerable problems for traditional NLP tools, which were developed with other domains in mind, which of- ten make strong assumptions about orthographic uniformity (i.e., there is just one way to spell you). One approach to cope with this problem is to an- notate in-domain data (Gimpel et al., 2011). Machine translation suffers acutely from the domain-mismatch problem caused by microblog text. On one hand, standard models are probably suboptimal since they (like many models) assume orthographic uniformity in the input. However, more acutely, the data used to develop these sys- tems and train their models is drawn from formal and carefully edited domains, such as parallel web pages and translated legal documents. MT training data seldom looks anything like microblog text. This paper introduces a method for finding nat- urally occurring parallel microblog text, which helps address the domain-mismatch problem. Our method is inspired by the perhaps surpris- ing observation that a reasonable number of mi- croblog users tweet “in parallel” in two or more languages. For instance, the American entertainer Snoop Dogg regularly posts parallel messages on Sina Weibo (Mainland China’s equivalent of Twit- ter), for example, watup Kenny Mayne!! - Kenny Mayne，最近这么样啊！！, where an English message and its Chinese translation are in the same post, separated by a dash. Our method is able to identify and extract such translations. Briefly, this requires determining if a tweet contains more than one language, if these multilingual …

Content truncated to 500 words