Enriching Biomedical Knowledge for Low-resource Language Through Large-scale Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3131–3142 May 2-6, 2023 ©2023 Association for Computational Linguistics Enriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translation Long Phan1∗, Tai Dang1,3∗, Hieu Tran1∗, Trieu H. Trinh1,4∗, Vy Phan3, Lam D. Chau2 and Minh-Thang Luong1 1VietAI Research 2Case Western Reserve University 3University of Massachusetts-Amherst 4New York University Abstract Biomedical data and benchmarks are highly valuable yet very limited in low-resource lan- guages other than English, such as Vietnamese. In this paper, we use a state-of-the-art transla- tion model in English-Vietnamese to translate and produce both pretrained and supervised data in the biomedical domains. Thanks to such large-scale translation, we introduce ViPub- medT5, a pretrained Encoder-Decoder Trans- former model trained on 20 million translated abstracts from the high-quality public PubMed corpus. ViPubMedT5 demonstrates state-of- the-art results on two different biomedical benchmarks in summarization and acronym disambiguation. Further, we release ViMedNLI - a new NLP task in Vietnamese translated from MedNLI using the recently public En-vi trans- lation model and carefully refined by human experts, with evaluations of existing methods against ViPubmedT5. 1 Introduction In recent years, pretrained language models (LMs) have played an important and novel role in devel- oping many Natural Language Processing (NLP) systems. Utilizing large pretrained models like BERT (Devlin et al., 2018), XLNET (Yang et al., 2019), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), GPT-3 (Brown et al., 2020) BART (Lewis et al., 2019), and T5 (Raffel et al., 2019) has become an effective trend in natural language pro- cessing. All these large models follow the Trans- former architecture proposed by (Vaswani et al., 2017) with the attention mechanism. The architec- ture has been proven to be very suitable for finetun- ing downstream tasks leveraging transfer learning with their large pretrained checkpoints. Before the emergence of large Transformer LMs, traditional wording embedding gave each word a fixed global *The first four authors contributed equally to this work representation. Large pretrained models can de- rive word vector representation from a trained large corpus. This will give the pretrained model a bet- ter knowledge of the generalized representation of a trained language/domain and significantly im- prove performance on downstream finetune tasks. The success of pretrained models on a generative domain (BERT, RoBERTa, BART, T5, etc.) has created a path in creating more specific-domain language models such as CodeBERT (Feng et al., 2020) and CoTexT (Phan et al., 2021b) for coding languages, TaBERT (Yin et al., 2020) for tabular data, BioBERT (Lee et al., 2019) and Pubmed- BERT (Tinn et al., 2021) for biomedical languages. Biomedical literature is getting more popular and widely accessible to the scientific community through large databases such as Pubmed1, PMC2, and MIMIC-IV (Johnson et al., 2021). This also leads to many studies, corpora, or projects released to further advance the Biomedical Natural Lan- guage Processing field (Lee et al., 2019; Tinn et al., 2021; Phan et al., 2021a; Yuan et al., 2022). These biomedical domain models …

Content truncated to 500 words