Lite Training Strategies for Portuguese-English and English-Portuguese Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 833–840 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 833 Lite Training Strategies for Portuguese-English and English-Portuguese Translation Alexandre Lopes1 Rodrigo Nogueira2,3,4 Roberto Lotufo2,3 Helio Pedrini1 1Institute of Computing, University of Campinas, Brazil 2School of Electrical and Computer Engineering, University of Campinas, Brazil 3NeuralMind Inteligˆencia Artificial, Brazil 4David R. Cheriton School of Computer Science, University of Waterloo, Canada Abstract Despite the widespread adoption of deep learn- ing for machine translation, it is still expensive to develop high-quality translation models. In this work, we investigate the use of pre-trained models, such as T5 (Raffel et al., 2019) for Portuguese-English and English-Portuguese translation tasks using low-cost hardware. We explore the use of Portuguese and English pre- trained language models and propose an adap- tation of the English tokenizer to represent Por- tuguese characters, such as diaeresis, acute and grave accents. We compare our models to the Google Translate API and MarianMT on a sub- set of the ParaCrawl dataset, as well as to the winning submission to the WMT19 Biomedi- cal Translation Shared Task. We also describe our submission to the WMT20 Biomedical Translation Shared Task. Our results show that our models have a competitive performance to state-of-the-art models while being trained on modest hardware (a single 8GB gaming GPU for nine days). Our data, models and code are available at https://github.com/ unicamp-dl/Lite-T5-Translation. 1 Introduction With the advent of deep neural networks, results in machine translation have recently improved over classical statistical strategies (Wu et al., 2016; Artetxe et al., 2018). For instance, in the Third and Fourth Conference on Machine Translation (WMT18 (Edunov et al., 2018) and WMT19 (Ng et al., 2019)), the top-performing systems for English-German and German-English competitions were based on transformers (Vaswani et al., 2017). Transformer models are state-of-the-art architec- tures for Machine Translation (MT) tasks and are capable of translating the same word to different words based on the context. For instance, the word ’bank’ in Portuguese can be translated to ’bench’ or ’bank’ depending on the context. This work explores translation strategies using language models pre-trained on Portuguese and English corpora. More specifically, we investi- gate the use of Text-to-Text Transfer Transformer (T5) pre-trained model for these tasks. T5 is a text-to-text Transformer trained with a similar masked language modeling objective as BERT. In this model, all target tasks are cast as sequence- to-sequence tasks. An illustration of T5 for the English-Portuguese translation task is shown in Figure 1. The main contributions of this work are: • We show that it is possible to train translation models that are competitive with the state of the art using few computational resources. We trained our models on a gaming desktop with an Nvidia RTX2070 GPU, i5 CPU, and 32GB RAM. In comparison, the winning submis- sion of the WMT19 Biomedical Translation Shared Task used four NVIDIA V100 GPUs, each being approximately ten times more ex- pensive than an RTX2070. • We created and made public ParaCrawl 99k, …

Content truncated to 500 words