Mukayese: Turkish NLP Strikes Back

Findings of the Association for Computational Linguistics: ACL 2022, pages 58 - 67 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation? En-Shiun Annie Lee,* Sarubi Thillainathan,† Shravan Nayak,‡ Surangika Ranathunga,† David Ifeoluwa Adelani,§,¶ Ruisi Su,∥ and Arya D. McCarthy# *University of Toronto, †University of Moratuwa, ‡IIT(BHU) Varanasi, §Masakhane NLP, ¶Saarland University, ∥Sway AI, #Johns Hopkins University annie.lee@cs.toronto.edu Abstract What can pre-trained multilingual sequence- to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology. In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems. While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU. In answer to our title’s question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data1. 1 Introduction Pre-trained multilingual sequence-to-sequence (PMSS) models, such as mBART (Tang et al., 2021) and mT5 (Xue et al., 2021), are pre-trained on large general data, then fine-tuned to deliver impressive results for natural language inference, question answering, and text simplification (Hu et al., 2020). Their performance on machine trans- lation shows promise for translating low-resource languages (Liu et al., 2021b; Adelani et al., 2021; Thillainathan et al., 2021), which remains an open challenge (Lopez and Post, 2013; Koehn and Knowles, 2017; Mager et al., 2021; Ranathunga et al., 2021). When can mBART and mT5 succeed in trans- lating a low-resource language? Despite their promise, the specific conditions for their practical application are not yet clear. Understanding their sensitivities is crucial to guide data acquisition ef- forts and apply PMSS models to new languages. 1Code is available at https://github.com/LRLNMT/ LRLNMT We introduce a framework for assessing data- dependency of performance of machine translation systems. We then apply it in a large-scale study of mBART’s viability for low-resource machine trans- lation on 10 typologically and geographically var- ied languages. Eight languages are low-resource, and four are unseen by mBART during pre-training. Through our results, we gauge the importance of five dimensions of the training data: 1. Amount of fine-tuning data 2. Noise in fine-tuning data 3. Amount of pre-training data 4. Domain mismatch 5. Language typology The closest work to ours (Liu et al., 2021b) consid- ers only the first two. For the seen languages, mBART reaches accept- able performance with either 10k high-quality, in- domain sentence pairs or 100k noisy ones. How- ever, mBART’s BLEU score for unseen languages is often below 3.0—far below usability. For these unseen, low-resource languages, the fact that even mBART—which has already seen billions of sentences—cannot succeed in virtually any of our conditions speaks to the need for appropriate in- domain data. Therefore, …

Content truncated to 500 words