CASIA’s System for IWSLT 2020 Open Domain Translation

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 130–139 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 130 CASIA’s System for IWSLT 2020 Open Domain Translation Qian Wang1,2, Yuchen Liu1,2, Cong Ma1,2, Yu Lu1,2, Yining Wang1,2, Long Zhou1,2 Yang Zhao1,2, Jiajun Zhang1,2, Chengqing Zong1,2 1National Laboratory of Pattern Recognition, CASIA, Beijing, China 2School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Abstract This paper describes the CASIA’s system for the IWSLT 2020 open domain transla- tion task. This year we participate in both Chinese→Japanese and Japanese→Chinese translation tasks. Our system is neural ma- chine translation system based on Transformer model. We augment the training data with knowledge distillation and back translation to improve the translation performance. Domain data classification and weighted domain model ensemble are introduced to generate the final translation result. We compare and analyze the performance on development data with differ- ent model settings and different data process- ing techniques. 1 Introduction Neural machine translation(NMT) has been in- troduced and made great success during the past few years (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Wu et al., 2016; Gehring et al., 2017; Zhou et al., 2017; Vaswani et al., 2017). Among those different neural network ar- chitectures, the Transformer, which is based on self-attention mechanism, has further improved the translation quality due to the ability of feature ex- traction and word sense disambiguation (Tang et al., 2018a,b). In this paper, we describe our Trans- former based neural machine translation system submitted to the IWSLT 2020 Chinese→Japanese and Japanese→Chinese open domain translation task (Ansari et al., 2020). Our system is built upon Transformer neural ma- chine translation architecture. We also adopt Rel- ative Position (Shaw et al., 2018) and Dynamic Convolutions (Wu et al., 2019) to investigate the performances of advanced model variations. For the implementation, we extend the latest release of Fairseq1 (Ott et al., 2019). 1https://github.com/pytorch/fairseq For data pre-processing, we use byte-pair encod- ing(BPE) segmentation (Sennrich et al., 2016b) for the source side and character level segmentation for the target side to improve the model performance on rare words. We also investigate the influence of different segmentation methods including word, BPE and character segmentation for both sides. To further improve the translation quality, we utilize data augmentation techniques of back- translation with a sub-selected monolingual corpus to build additional pseudo parallel training data. Sentence level knowledge distillation is used to strengthen the performance of student model from multi-policy teacher models including left→right, right→left, source→target and target→source. We also investigate the domain information of the large training data by using a Bert based do- main classifier, which is a masked language model and has been shown effective in large scale text classification tasks (Devlin et al., 2019). With the in-domain data, we transfer the model of general domain to each specific domain, and use weighted domain model ensemble as decoding strategy. 2 System Description Figure 1 depicts the whole process of our submis- sion …

Content truncated to 500 words