A Neural Transition-based Joint Model for Disease Named Entity Recognition and Normalization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 244–258 August 1–6, 2021. ©2021 Association for Computational Linguistics 244 Contrastive Learning for Many-to-many Multilingual Neural Machine Translation Xiao Pan, Mingxuan Wang, Liwei Wu, Lei Li ByteDance AI Lab {panxiao.94,wangmingxuan.89,wuliwei.000,lileilab}@bytedance.com Abstract Existing multilingual machine translation ap- proaches mainly focus on English-centric di- rections, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an em- phasis on the quality of non-English language directions. Our intuition is based on the hy- pothesis that a universal cross-language rep- resentation leads to better multilingual trans- lation performance. To this end, we pro- pose mRASP2, a training method to ob- tain a single unified multilingual translation model. mRASP2 is empowered by two tech- niques: a) a contrastive learning scheme to close the gap among representations of dif- ferent languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 out- performs existing best unified model and achieves competitive or even better perfor- mance than the pre-trained and fine-tuned model mBART on tens of WMT’s transla- tion directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github. com/PANXiao1994/mRASP2. 1 Introduction Transformer (Vaswani et al., 2017) has achieved decent performance for machine translation with rich bilingual parallel corpora. Recent work on multilingual machine translation aims to create a single unified model to translate many lan- guages (Johnson et al., 2017; Aharoni et al., 2019; Zhang et al., 2020; Fan et al., 2020; Siddhant et al., 2020). Multilingual translation models are appeal- ing for two reasons. First, they are model effi- cient, enabling easier deployment (Johnson et al., Encoder Decoder <Fr> Je t’aime. <Fr> C’est la vie. …… <Zh> ��� <En> It’s sunny. <En> I love you. … Anchor + <Fr> Je t’aime. Lctr — Positive Negative Lce Figure 1: The proposed mRASP2. It takes a pair of par- allel sentences (or augmented pseudo-pair) and com- putes normal cross entropy loss with a multi-lingual encoder-decoder. In addition, it computes contrastive loss on the representations of the aligned pair (positive example) and randomly selected non-aligned pair (neg- ative example). 2017). Further, parameter sharing across different languages encourages knowledge transfer, which benefits low-resource translation directions and potentially enables zero-shot translation (i.e. di- rect translation between a language pair not seen during training) (Ha et al., 2017; Gu et al., 2019; Ji et al., 2020). Despite these benefits, challenges still remain in multilingual NMT. First, previous work on mul- tilingual NMT does not always perform well as their corresponding bilingual baseline especially on rich resource language pairs (Tan et al., 2019; Zhang et al., 2020; Fan et al., 2020). Such per- formance gap becomes larger with the increasing number of accommodated languages for multilin- gual NMT, as model capacity necessarily must …

Content truncated to 500 words