Multilingual and Multimodal Learning for Brazilian Portuguese

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 919–927 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 919 Multilingual and Multimodal Learning for Brazilian Portuguese Júlia Sato∗, Helena Caseli∗, Lucia Specia† ∗Federal University of São Carlos (UFSCar) Rod. Washington Luiz, s/n, São Carlos, Brazil juliasato@estudante.ufscar.br, helenacaseli@ufscar.br †Imperial College London Exhibition Rd, South Kensington, London SW7 2BX, United Kingdom l.specia@ic.ac.uk Abstract Humans constantly deal with multimodal information, that is, data from different modalities, such as texts and images. In order for machines to process information similarly to humans, they must be able to process multimodal data and understand the joint relationship between these modalities. This paper describes the work performed on the VTLM (Visual Translation Language Modelling) framework from (Caglayan et al., 2021) to test its generalization ability for other language pairs and corpora. We use the multimodal and multilingual corpus How2 (Sanabria et al., 2018) in three parallel streams with aligned English-Portuguese-Visual information to investigate the effectiveness of the model for this new language pair and in more complex scenarios, where the sentence associated with each image is not a simple description of it. Our experiments on the Portuguese-English multimodal translation task using the How2 dataset demonstrate the efficacy of cross-lingual visual pretraining. We achieved a BLEU score of 51.8 and a METEOR score of 78.0 on the test set, outperforming the MMT baseline by about 14 BLEU and 14 METEOR. The good BLEU and METEOR values obtained for this new language pair, regarding the original English-German VTLM, establish the suitability of the model to other languages. Keywords: Multilingual Language Model, Multimodal Machine Translation, Brazilian Portuguese, Vision and Lan- guage 1. Introduction Understanding different modalities together is an im- portant aspect of human comprehension. We often use sight, hearing, smell and other senses to assimilate a single concept. This multimodal aspect of learning can be very useful for machines to process multimodal information and understand the joint relationship be- tween these modalities. While multimodal models are trained to be able to in- terpret and associate data from different modalities – such as text, audio and image simultaneously – multi- lingual models are meant to understand multiple lan- guages by learning cross-lingual representations. In this context, models that learn multimodal and mul- tilingual representations have been shown to perform better in many natural language tasks (Caglayan et al., 2021). Recently, the area of Natural Language Processing (NLP) has been experiencing a significant paradigm shift with the proposition of several neural models (deep learning) for language processing (Devlin et al., 2019; Conneau and Lample, 2019; Radford et al., 2018). These advances are based on the use of artificial neural networks and strategies such as transfer learn- ing and attention (Calixto et al., 2016; Caglayan et al., 2016; Libovický and Helcl, 2017). There are several examples of NLP applications for which these strategies have reached state of the art in monolingual (Lan et al., 2020; Liu et al., 2019; Rothe et al., 2020), …

Content truncated to 500 words