The Multilingual Microblog Translation Corpus: Improving and Evaluating Translation of User-Generated Text

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 910–918 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 910 The Multilingual Microblog Translation Corpus: Improving and Evaluating Translation of User-Generated Text Paul McNamee and Kevin Duh Johns Hopkins University Human Language Technology Center of Excellence 810 Wyman Park Drive, Baltimore MD 21211, USA mcnamee@jhu.edu, kevinduh@cs.jhu.edu Abstract Translation of the noisy, informal language found in social media has been an understudied problem, with a principal factor being the limited availability of translation corpora in many languages. To address this need we have developed a new corpus containing over 200,000 translations of microblog posts that supports translation of thirteen languages into English. The languages are: Arabic, Chinese, Farsi, French, German, Hindi, Korean, Pashto, Portuguese, Russian, Spanish, Tagalog, and Urdu. We are releasing these data as the Multilingual Microblog Translation Corpus to support further research in translation of informal language. We establish baselines using this new resource, and we further demonstrate the utility of the corpus by conducting experiments with fine-tuning to improve translation quality from a high performing neural machine translation (NMT) system. Fine-tuning provided substantial gains, ranging from +3.4 to +11.1 BLEU. On average, a relative gain of 21% was observed, demonstrating the utility of the corpus. Keywords: Machine Translation, Informal Language, Domain Adaptation, Evaluation Corpora 1. Introduction Large collections of parallel text, or bitext, are increas- ingly available in many languages and in ever grow- ing quantities. One well known project is OPUS, the Open Parallel Corpus, a web portal containing many open source parallel corpora which is maintained by J¨org Tiedemann (2012). It is hard to overstate the im- pact that comes from the wide availability of such data; it is now possible to build translation systems for lan- guage pairs that might have been infeasible just a few years ago. However, the largest sources of parallel data continue to be specialized domains such as governmen- tal documents (e.g., European Parliament) or religious texts. This creates an issue of domain mismatch, where the sources of training data are frequently unlike the types of data being translated by end users. One genre where this is problematic is in user- generated texts found in online social media platforms (e.g., Facebook, Reddit, and Twitter). Unlike copy- edited news, user-produced text has a much higher spelling error rate. The language used on these plat- forms is often vernacular vs. formal, and contains a high proportion of abbreviations and other shorthand notations. There are many peer-to-peer messages writ- ten in the first or second person point of view, unlike news or reports written in the third person. There is a higher percentage of non-linguistic or quasi-linguistic content such as user IDs, URLs, hashtags, and emoji. And the use of case and punctuation is often atypi- cally applied for stylistic effect. These issues and many others contribute to domain mismatch, and as a re- sult translation performance is degraded when a model trained on standard datasets …

Content truncated to 500 words