Synthesizing Parallel Data of User-Generated Texts with Zero-Shot Neural Machine Translation

Synthesizing Parallel Data of User-Generated Texts with Zero-Shot Neural Machine Translation Benjamin Marie Atsushi Fujita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {bmarie, atsushi.fujita}@nict.go.jp Abstract Neural machine translation (NMT) systems are usually trained on clean parallel data. They can perform very well for translating clean in-domain texts. However, as demonstrated by previous work, the translation quality signifi- cantly worsens when translating noisy texts, such as user-generated texts (UGT) from on- line social media. Given the lack of parallel data of UGT that can be used to train or adapt NMT systems, we synthesize parallel data of UGT, exploiting monolingual data of UGT through crosslingual language model pre- training and zero-shot NMT systems. This paper presents two different but complemen- tary approaches: One alters given clean parallel data into UGT-like parallel data whereas the other generates translations from monolingual data of UGT. On the MTNT translation tasks, we show that our synthesized parallel data can lead to better NMT systems for UGT while making them more robust in translating texts from various domains and styles. 1 Introduction Neural machine translation (NMT) requires large parallel data for training. However, even when trained on large clean parallel data, NMT gen- erates translations of very poor quality when translating out-of-domain or noisy texts. For instance, Michel and Neubig (2018) empirically showed that NMT systems trained on clean parallel data from the news and parliamentary debate domains perform reasonably well when translating news articles but poorly perform at translating user-generated texts (UGT) from a social media. UGT can be from various domains and manifest various forms of natural noise. For instance, they can exhibit spelling/typographical errors, words omission/insertion/repetition, gram- matical/syntactic errors, or noise markers even more specific to the writing style of social media such as abbreviations, obfuscated profani- ties, inconsistent capitalization, Internet slang, and emojis. Normalizing and correcting them in a pre- processing step is a solution to facilitate translation (Gerlach et al., 2013; Matos Veliz et al., 2019), but it impedes the correct transfer of the style of the source text to its translation. In this paper, we posit that the NMT system should preserve the style during the translation. Another trend of work focuses on making NMT more robust in handling noisy tokens, such as tokens with spelling mis- takes, which can greatly disturb NMT (Belinkov and Bisk, 2018). However, it has only a mini- mal impact in translating UGT (Karpukhin et al., 2019) that contains other types of noise/errors. Whereas domain adaptation methods are help- ful in improving NMT for UGT (Li et al., 2019), we do not usually have bilingual parallel data of UGT created by professional translators to train or adapt an NMT system. Consequently, previous work on NMT for UGT merely focused on sce- narios for which we have UGT parallel data, such as the MTNT dataset (Michel and Neubig, 2018). In contrast to previous work, we assume that parallel data of UGT are not available and that we can only …

Content truncated to 500 words