Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling

Proceedings of the 5th Conference on Machine Translation (WMT), pages 100–104 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 100 GTCOM Neural Machine Translation Systems for WMT20 Chao Bei, Hao Zong, Qinming Liu and Conghu Yuan Global Tone Communication Technology Co., Ltd. {beichao, zonghao, liuqingmin and yuanconghu}@gtcom.com.cn Abstract This paper describes the Global Tone Commu- nication Co., Ltd.’s submission of the WMT20 shared news translation task. We partici- pate in four directions: English to (Khmer and Pashto) and (Khmer and Pashto) to En- glish. Further, we get the best BLEU scores in the directions of English to Pashto, Pashto to English and Khmer to English (13.1, 23.1 and 25.5 respectively) among all the partic- ipants. Our submitted systems are uncon- strained and focus on mBART (Multilingual Bidirectional and Auto-Regressive Transform- ers), back-translation and forward-translation. Also, we apply rules, language model and RoBERTa model to filter monolingual, paral- lel sentences and synthetic sentences. Besides, we validate the difference of the vocabulary built from monolingual data and parallel data. 1 Introduction We participated in the WMT shared news transla- tion task and focus on the bidirections: English and Khmer, English and Pashto. We applied fairseq(Ott et al., 2019) as our develop tool and use trans- former(Vaswani et al., 2017) as the main archi- tecture. The primary ranking index for submitted systems is BLEU (Papineni et al., 2002), therefore we apply BLEU as the evaluation matrix for our translation system. For Khmer, we use polyglot 1 as the tokenizer before evaluation. For data preprocessing, the basic method in- cludes punctuation normalization for all language. Further, according to the different language charac- teristics. Tokenization, truecase and byte pair en- coding (BPE) (Sennrich et al., 2015b) are applied for English, and sentencepiece (Kudo and Richard- son, 2018) is applied for Khmer and Pashto. Be- sides, human rules, language model and RoBERTa model (Liu et al., 2019) are also involved to clean 1https://github.com/aboSamoor/polyglot parallel data, monolingual data and synthetic data. Regard to the techniques on model training, back- translation (Sennrich et al., 2015a) and forward- translation are applied to verify whether these tech- niques could improve the translation performance especially in low-resource condition. We all know that it is more difficult to train a model in low-resource condition, because it suffers from data sparsity and out-of-vocabulary problem. Normally knowledge distillation (Kim and Rush, 2016) is a good way to generate synthetic data. But in this task we suppose that knowledge distillation can only generate 100 thousand to 1 million par- allel sentences due to the size of provided data. Therefore, we use forward-translation with mono- lingual data to generate more synthetic data. Here forword-translation refers to translate the source sentences to target language, and clean synthetic data. This paper is arranged as follows. We firstly describe the task and show the data information, then introduce how we do data filtering, includ- ing human rules, language model and RoBERTa model. After that, we describe the techniques on low-resource condition and show the conducted ex- periments in …

Content truncated to 500 words