Small Model and In-Domain Data Are All You Need

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 255–259 November 10–11, 2021. ©2021 Association for Computational Linguistics 255 Abstract I participated in the WMT shared news translation task and focus on one high resource language pair: English and Chinese (two directions, Chinese to English and English to Chinese). The submitted systems (ZengHuiMT) focus on data cleaning, data selection, back translation and model ensemble. The techniques I used for data filtering and selection include filtering by rules, language model and word alignment. I used a base translation model trained on initial corpus to obtain the target versions of the WMT21 test sets, then I used language models to find out the monolingual data that is most similar to the target version of test set, such monolingual data was then used to do back translation. On the test set, my best submitted systems achieve 35.9 and 32.2 BLEU for English to Chinese and Chinese to English directions respectively, which are quite high for a small model. 1 Introduction I participated in the WMT shared news translation task and focus on the English and Chinese language pair. This language pair is challenging due to the plentiful in-domain bitext training data and abundant monolingual data. High resource means fierce competition, many high-tech companies and universities chose this language pair also. My neural machine translation system is developed using base transformer (Vaswani et al., 2017) architecture and the toolkit I used is THUMT (Zhang et al., 2020). Rules and word aligning model are used to clean parallel data. Language model is used to clean monolingual 1http://mteval.cipsc.org.cn:81/agreement/description data. I use a base transformer (Vaswani et al., 2017) architecture since I have only one GPU. The following techniques are used on model training: a. Increase the number of encoder layers to 12 to further improve the encoder’s representation capability; b. Back translation (Sennrich et al., 2016) are applied to fully utilize the monolingual corpus. c. Shared vocabulary is used for better performance. d. Four different models using diversified data are trained for ensemble decoding. 2 Data Filtering and Selection The parallel data is mainly from CCMT Corpus1, and the monolingual data is collected from the internet. I did not use any other datasets since I think they are not highly related to this news translation task. To evaluate my model’s performance, I merged the test set from WMT2017 to WMT2020 to build a big development set. 2.1 Monolingual Data Filtering Using Language Model In terms of monolingual data, I collected more than 20 million Chinese sentences and more than 15 million English sentences from various websites. The Chinese text are collected from the following websites: http://www.chinanews.com/ https://cn.reuters.com/ http://news.ifeng.com/ http://people.com.cn/ https://www.sina.com.cn/ http://www.xinhuanet.com/ https://news.cctv.com/ https://www.qq.com/ Small Model and In-Domain Data are All You Need Hui Zeng Independent Researcher felix_zeng_ai@aliyun.com 256 https://www.sohu.com/ The English text are collected from the following websites: https://www.bbc.com/news www.theguardian.com https://www.telegraph.co.uk/ https://www.washingtonpost.com/ https://www.latimes.com/ https://www.smh.com.au/ https://www.brisbanetimes.com.au/ https://www.dailymail.co.uk/ https://www.rt.com/ https://time.com/ https://www.euronews.com/ https://www.msnbc.com/ https://www.cbsnews.com/ https://www.nytimes.com/ https://www.newsweek.com/ https://www.foxnews.com/ https://www.standard.co.uk/ The following rules are used for a simple cleaning: •Remove …

Content truncated to 500 words