TweetMT: A Parallel Microblog Corpus

TweetMT: A parallel microblog corpus I˜naki San Vicente1, I˜naki Alegr´ıa2, Nora Aranberri2, Cristina Espa˜na-Bonet3, Pablo Gamallo4, Hugo Gonc¸alo Oliveira5, Eva Mart´ınez2, Antonio Toral6, Arkaitz Zubiaga7 1 Elhuyar, 2 University of the Basque Country, 3 Universitat Polit`ecnica de Catalunya, 4 Universidade de Santiago de Compostela, 5 University of Coimbra, 6 Dublin City University, 7 University of Warwick tweetmt@elhuyar.com Abstract We introduce TweetMT, a parallel corpus of tweets in four language pairs that combine five languages (Spanish from/to Basque, Catalan, Galician and Portuguese), all of which have an official status in the Iberian Peninsula. The corpus has been created by combining automatic collection and crowdsourcing approaches, and it is publicly available. It is intended for the development and testing of microtext machine translation systems. In this paper we describe the methodology followed to build the corpus, and present the results of the shared task in which it was tested. Keywords: Machine Translation, Microblogs, Tweets, Social Media, parallel corpus 1. Introduction While machine translation (MT) is a mature research field now, the application of MT techniques to tweets is still in its infancy. Tweets are often written from mobile devices, which exacerbates the poor quality of the spelling, and include linguistic inaccuracies, symbols and diacritics. Tweets also vary in terms of structure, including features which are exclusive to the platform, such as hashtags, user mentions, and retweets. These characteristics make the application of MT to tweets a new challenge that requires specific processing techniques to perform effectively. Despite the paucity of research in the specific task of translating tweets, an increasing interest can be observed in the scientific community (Gotti et al., 2013; Peisenieks and Skadin¸ˇs, 2014). Similarly, a related and highly relevant direction of research is the work on MT of SMS texts, such as Munro’s study in the context of the 2010 Haiti earthquake (Munro, 2010). Provided the dearth of benchmark resources and comparison studies bringing to light the potential and shortcomings of today’s MT techniques applied to tweets, a corpus was compiled in the framework of TweetMT, a workshop and shared task1 on MT applied to tweets. Our parallel corpus includes tweets for the following language pairs: Catalan–Spanish (ca-es), Basque–Spanish (eu-es), Galician–Spanish (gl-es), and Portuguese–Spanish (pt-es). Those are the most common pairings between official languages in the Iberian Peninsula. 2. Collecting parallel tweets To the best of our knowledge, there is no parallel tweet dataset available apart from that produced by (Ling et al., 2013), which differs from our purposes in that they worked on tweets that mix two languages, i.e., providing the translated text within the same tweet. They further improve the quality of the parallel segments by means of crowdsourced annotations (Ling et al., 2014). Since we 1http://komunitatea.elhuyar.eus/tweetmt/ wanted to work on the translation of entire tweets into new tweets, we generated a corpus for the specific purposes of the TweetMT Workshop. For corpus generation, we developed a semi-automatic method to retrieve and align parallel tweets. The first step of this method consists in identifying multiple Twitter authors …

Content truncated to 500 words