MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11445–11465 July 9-14, 2023 ©2023 Association for Computational Linguistics MULTIINSTRUCT: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning Zhiyang Xu∗, Ying Shen∗, Lifu Huang Computer Science Department Virginia Tech {zhiyangx, yings, lifuh}@vt.edu Abstract Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multi- modal tasks. In this work, we introduce MUL- TIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to- seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert- written instructions. We take OFA (Wang et al., 2022a) as the base pre-trained model for mul- timodal instruction tuning, and to further im- prove its zero-shot performance, we explore multiple transfer learning strategies to lever- age the large-scale NATURAL INSTRUCTIONS dataset (Mishra et al., 2022). Experimental re- sults demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new eval- uation metric – Sensitivity, to evaluate how sen- sitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instruc- tions for each task1. 1 Introduction With the advances in large-scale pre-trained lan- guage models (PLMs), recent studies have explored various efficient learning paradigms (Brown et al., 2020; Liu et al., 2021; Wei et al., 2021; Xie et al., 2021) to generalize PLMs to new tasks without task-specific tuning. Among these, instruction work.∗ Zhiyang Xu and Ying Shen contributed equally to this 1The dataset, source code, and model checkpoints are publicly available at https://github.com/VT-NLP/ MultiInstruct. tuning (Wei et al., 2021) has achieved significant success in zero-shot learning on natural language processing tasks. By fine-tuning a PLM on tasks described through instructions, instruction tuning allows the model to learn to understand and fol- low the instructions to perform predictions on un- seen tasks. Recent advancement in multimodal pre- training (Wang et al., 2022a; Alayrac et al., 2022; Bao et al., 2022; Wang et al., 2022c) has shown the potential of jointly interpreting text and images in a shared semantic space, which further leads us to ask: can the instruction tuning be leveraged to im- prove the generalizability of Vision-Language pre- trained models on multi-modal and vision tasks? In this work, we propose MULTIINSTRUCT, the first benchmark dataset for multimodal in- struction tuning with 62 diverse tasks from 10 broad categories, including Visual Question An- swering (Goyal et al., 2017; Suhr et al., 2017), Commonsense Reasoning (Zellers et al., 2019; Xie et al., 2019), Visual Relationship Understand- ing (Krishna et al., 2017) and so on. We equipped each task with …

Content truncated to 500 words