BERTGen: Multi-task Generation through BERT

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6440–6455 August 1–6, 2021. ©2021 Association for Computational Linguistics 6440 BERTGEN: Multi-task Generation through BERT Faidon Mitzalis1, Ozan Caglayan1, Pranava Madhyastha1, Lucia Specia1,2 1Department of Computing, Imperial College London, UK 2Department of Computer Science, University of Sheffield, UK phaedonmit@gmail.com, {o.caglayan,pranava,lspecia}@ic.ac.uk Abstract We present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pre- trained models VL-BERT and M-BERT, re- spectively. BERTGEN is auto-regressively trained for language generation tasks, namely image captioning, machine translation and multimodal machine translation, under a multi- task setting. With a comprehensive set of evaluations, we show that BERTGEN outper- forms many strong baselines across the tasks explored. We also show BERTGEN’s abil- ity for zero-shot language generation, where it exhibits competitive performance to super- vised counterparts. Finally, we conduct abla- tion studies which demonstrate that BERTGEN substantially benefits from multi-tasking and effectively transfers relevant inductive biases from the pre-trained models. 1 Introduction Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU), resulting in high performance ceilings across multiple tasks (Devlin et al., 2019; Yang et al., 2019; Dong et al., 2019). The recent success of language model pre-training with masked language modelling (MLM) such as BERT (Devlin et al., 2019) further paved the way for more complex approaches that combine lan- guage pre-training with images (Tan and Bansal, 2019; Su et al., 2020; Lu et al., 2020), video (Sun et al., 2019), and speech (Chuang et al., 2020). Most of these approaches follow a task-specific fine-tuning step after the model is pre-trained. However, there has been little work on exploiting pre-trained MLMs for natural language generation (NLG) tasks. Previous work argues that the MLM objective is ill-suited for generation tasks such as machine translation (Yang et al., 2019; Rothe et al., 2020). Recent work in this direction has predom- inantly investigated the use of pre-trained mod- els to either initialise Transformer-based encoder- decoder models (Imamura and Sumita, 2019; Clin- chant et al., 2019; Yang et al., 2020; Rothe et al., 2020) or to distill knowledge for sequence genera- tion tasks (Chen et al., 2020). In this work, we present BERTGEN, which ex- tends BERT in a generative setting (§ 2.1). This re- sults in a single generator – without a separation be- tween the encoder and the decoder – capable of con- suming multiple input modalities and generating in multiple languages. The latter features are achieved by transferring knowledge from state-of-the-art pre- trained models, namely VL-BERT (Su et al., 2020) and multilingual BERT (M-BERT) (Devlin et al., 2019). We train BERTGEN on various tasks, in- cluding image captioning, machine translation and multimodal machine translation, and datasets in four different languages (§ 2.2). Based on a number of experiments, our find- ings (§ 3) show that BERTGEN (i) is surprisingly versatile as it is capable of describing images and performing translation in …

Content truncated to 500 words