Deep Blue Sonics’ Submission to IWSLT 2020 Open Domain Translation Task

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 140–144 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 140 Deep Blue Sonics’ Submission to IWSLT 2020 Open Domain Translation Task Enmin Su Deep Blue Sonics enmin.su@pku.edu.cn Yi Ren Deep Blue Sonics yi.ren@dbsonics.net Abstract We present in this report our submission to IWSLT 2020 Open Domain Translation Task(Ansari et al., 2020). We built a data pre- processing pipeline to efficiently handle large noisy web-crawled corpora, which boosts the BLEU score of a widely used transformer model in this translation task. To tackle the open-domain nature of this task, back- translation (Sennrich et al., 2016) is applied to further improve the translation performance. 1 Introduction Neural machine translation (NMT) is a well- studied problem boosted in recent years by pow- erful transformer models (Vaswani et al., 2017), which find their ways to various sequence-to- sequence tasks, such as automatic speech recog- nition (ASR) (Dong et al., 2018), speech transla- tion (Gangi et al., 2019), text-to-speech (Shin et al., 2019), to name a few. Nevertheless, many chal- lenges remain in training an efficient transformer NMT model in practice, such as the following ones raised in IWSLT 2020 Open Domain Translation Task: Handling noisy dataset Hassan et al. pointed out that machine translation models are vulnerable to noise even in small quantity. In practice, manual correction of a massive corpora is prohibitive, thus calling for an automatic data cleaning pipeline. Leveraging monolingual data Compared to parallel corpora, monolingual data can be acquired at a much lower cost. Common ways of us- ing monolingual data include language modeling (C¸ aglar G¨ulc¸ehre et al., 2015), back-translation (Sennrich et al., 2016), and dual learning (He et al., 2016), all exhibiting promising results; further- more, they could be adopted in a complementary way when carefully designed (Hassan et al., 2018). Last but not least, massive pre-trained language models like BERT perform strongly in NLP tasks like question answering, reading comprehension and text classification (Devlin et al., 2019), motivat- ing our attempts to incorporate them in our NMT system. Domain mismatch NMT systems trained with data from specific domains may translate poorly in other domains (Freitag and Al-Onaizan, 2016). Training the model with all available corpora, and fine-tuning it on a specific domain generally achieves best results in this domain (Chu et al., 2017). In open domain cases, it’s impractical to keep a dedicated model or to obtain enough training data for every single domain. Hence Multi- Domain NMT, where a single model generalizes to multiple domains, is gaining interest in recent research. For example, Tars and Fishel; Jiang et al.; Zeng et al. injected domain information into model input, leading to convincing and consistent improvements, in which domain information may be derived in both supervised and unsupervised manners. Our work consists of establishing an efficient data pre-processing pipeline for large web-crawled corpora to train a transformer model for NMT and exploiting large amount of monolingual data with back-translation and language modeling. This report is …

Content truncated to 500 words