MIMOQA: Multimodal Input Multimodal Output Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498 June 6–11, 2021. ©2021 Association for Computational Linguistics 483 mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer Linting Xue∗ Noah Constant∗ Adam Roberts∗ Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel Google Research Abstract The recent “Text-to-Text Transfer Trans- former” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art re- sults on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset cover- ing 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilin- gual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its pre- diction into the wrong language. All of the code and model checkpoints used in this work are publicly available.1 1 Introduction Current natural language processing (NLP) pipelines often make use of transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on a downstream task of interest (Ruder et al., 2019). The success of this paradigm is partially thanks to the release of parameter check- points for pre-trained models. These checkpoints allow members of the NLP community to quickly attain strong performance on many tasks without needing to perform expensive pre-training them- selves. As one example, the pre-trained check- points for the “Text-to-Text Transfer Transformer” (T5) model released by Raffel et al. (2020) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al., 2020; Roberts et al., 2020; Kale, 2020; Izacard and Grave, 2020; Nogueira et al., 2020; Narang et al., 2020, etc.). Unfortunately, many of these language models were pre-trained solely on English-language text. ∗Equal Contribution. Please direct correspondence to lintingx@google.com, nconstant@google.com, adarob@google.com, and craffel@google.com 1https://goo.gle/mt5-code This significantly limits their use given that roughly 80% of the world population does not speak En- glish (Crystal, 2008). One way the community has addressed this English-centricity has been to release dozens of models, each pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Tuan Nguyen, 2020; Polignano et al., 2019, etc.). A more general solution is to produce multilingual models that have been pre-trained on a mixture of many languages. Popular models of this type are mBERT (Devlin, 2018), mBART (Liu et al., 2020a), and XLM-R (Conneau et al., 2020), which are multilingual variants of BERT (Devlin et al., 2019), BART (Lewis et al., 2020b), and RoBERTa (Liu et al., 2019), respectively. In this paper, we continue this tradition by re- leasing mT5, a multilingual variant of T5. Our goal with mT5 is to produce a massively multilingual model that deviates as little as possible from the recipe used to …

Content truncated to 500 words