Manifold’s English-Chinese System at WMT22 General MT Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 275 - 279 December 7-8, 2022 ©2022 Association for Computational Linguistics Manifold’s English-Chinese System at WMT22 General MT Task Chang Jin Soochow University∗ cjin@stu.suda.edu.cn Tingxun Shi OPPO shitingxun@oppo.com Zhengshan Xue OPPO xuezhengshan@oppo.com Xiaodong Lin Rutgers University lin@business.rutgers.edu Abstract Manifold’s English-Chinese System WMT22 is an ensemble of 4 models trainedat by different configurations with scheduled sampling-based fine-tuning. The four config- urations are DeepBig (XenC), DeepLarger (XenC), DeepBig-TalkingHeads (XenC) and DeepBig (LaBSE). Concretely, DeepBig extends Transformer-Big to 24 encoder layers. DeepLarger has 20 encoder layers and its feed-forward network (FFN) dimension is 8192. TalkingHeads applies the talking-heads trick. For XenC configs, we selected mono- lingual and parallel data that is similar to the past newstest datasets using XenC, and for LaBSE, we cleaned the officially provided parallel data using LaBSE pretrained model. According to the officially released autonomic metrics leaderboard1, our final constrained system ranked 1st among all others when evaluated by bleu-all, chrf-all and COMET-B, 2nd by COMET-A. 1 Introduction This report describes Manifold’s machine translation system submitted to WMT 22 English→Chinese general domain translation task. The general domain translation task of WMT is a new task set up this year, replaces the time-honored news translation task. However, as the newstest datasets released previously were created by professional translators manually, they were considered to have better quality compared with the web crawled dataset. Therefore, generally our strategy is selecting data from the provided datasets according to their similarity between past WMT test sets, and training big models with different architectures or training methods based on the selected data. ∗Work done during Jin’s internship in OPPO. 1https://github.com/wmt-conference/ wmt22-news-systems/blob/main/scores/ automatic-scores.tsv 2 Data Preprocessing For officially released bilingual data, we merged ParaCrawl v9, News Commentary v16, Wiki Titles v3, UN Parallel Corpus v1.0, CCMT corpus and WikiMatrix corpus together, then applied multiple rules to preprocess and filter the officially released bilingual data, including • Normalizing punctuation. • Removing unprintable characters. • Tokenizing texts. jieba2 was applied for Chi- nese texts, and moses tokenizer3 was applied for English texts. • Removing all the sentence pairs whose source text or target text tokens count exceeds 150, and the segment pairs with a source-target token ratio lower than 2/3 or higher than 3/2. • Removing all the sentence pairs that belong to other languages. We applied the fasttext model (Joulin et al., 2016)4 as our language identifier. • Deduplicating the dataset. 3 Basic models After applying the aforementioned preprocessing and filtering rules, 30M segment pairs were kept 5. For the kept data, we adopted two different selection strategies: 1. Following NiuTrans’ system submitted to WMT21 (Zhou et al., 2021), we selected 12M segment pairs from the kept data which are 2https://github.com/fxsjy/jieba 3https://github.com/moses-smt/mosesdecoder/ blob/master/scripts/tokenizer/tokenizer.perl 4https://dl.fbaipublicfiles.com/fasttext/ supervised-models/lid.176.bin 5We did not make use of official back-translated corpus 275 Config Item Big Deep Deeper DeepBig DeepLarger # Encoder Layer 6 30 40 24 20 # Attention Heads 16 8 8 16 16 Embedding Size 1024 512 512 1024 1024 FFN Size …

Content truncated to 500 words