Leveraging Highly Accurate Word Alignment for Low Resource Translation by Pretrained Multilingual Model

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 336–347 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 336 Leveraging Highly Accurate Word Alignment for Low Resource Translation by Pretrained Multilingual Model Jingyi Zhu s2120801 @ u.tsukuba.ac.jp Minato Kondo s2320743 @ u.tsukuba.ac.jp Takuya Tamura s2120744 @ u.tsukuba.ac.jp Takehito Utsuro utsuro @ iit.tsukuba.ac.jp Deg. Prog. Sys.&Inf. Eng., Grad. Sch. Sci.&Tech., University of Tsukuba, Japan Masaaki Nagata masaaki.nagata @ ntt.com NTT Communication Science Laboratories, NTT Corporation, Japan Abstract Recently, there has been a growing interest in pretraining models in the field of natural lan- guage processing. As opposed to training models from scratch, pretrained models have been shown to produce superior results in low-resource translation tasks. In this paper, we intro- duced the use of pretrained seq2seq models for preordering and translation tasks. We utilized manual word alignment data and mBERT-based generated word alignment data for training preordering and compared the effectiveness of various types of mT5 and mBART models for preordering. For the translation task, we chose mBART as our baseline model and evaluated several input manners. Our approach was evaluated on the Asian Language Treebank dataset, consisting of 20,000 parallel data in Japanese, English and Hindi, where Japanese is either on the source or target side. We also used in-house 3,000 parallel data in Chinese and Japanese. The results indicated that mT5-large trained with manual word alignment achieved a preorder- ing performance exceeding 0.9 RIBES score on Ja-En and Ja-Zh pairs. Moreover, our proposed approach significantly outperformed the baseline model in most translation directions of Ja-En, Ja-Zh, and Ja-Hi pairs in at least one of BLEU/COMET scores. 1 Introduction In recent years, there has been a growing body of research on sequence-to-sequence (seq2seq) models that are based on pretraining (Xue et al., 2021; Liu et al., 2020; Lin et al., 2020). Since the introduction of the Transformer architecture (Vaswani et al., 2017), the quality of machine translation has greatly improved. However, when it comes to low-resource translation tasks, the performance of this type of parameter randomization model often suffers due to the limited size of available datasets (Sennrich and Zhang, 2019; Lee et al., 2022; Zhu et al., 2022). To address this challenge, many researchers have proposed using unsupervised meth- ods, such as mapping monolingual vector embeddings to a common cross-lingual embedding space (Lin et al., 2020; Sen et al., 2019), or leveraging large-scale pretraining models that have been successfully applied to various NLP tasks (Devlin et al., 2019; Brown et al., 2020). 337 In this paper, we propose applying a pretrained seq2seq model for preordering and trans- lation tasks. Specifically, we investigate different sizes of mT5s (Xue et al., 2021) and mBART (Liu et al., 2020), in order to evaluate their performance on preordering when using manual word alignment data. For the translation process, we choose mBART as our baseline model, and we evaluate the translation results using both the original …

Content truncated to 500 words