FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8158–8185 July 9-14, 2023 ©2023 Association for Computational Linguistics FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning Qinyuan Ye1† Iz Beltagy2 Matthew E. Peters2 Xiang Ren1,2 Hannaneh Hajishirzi2,3 1University of Southern California 2Allen Institute for AI 3University of Washington {qinyuany,xiangren}@usc.edu {beltagy,matthewp,hannah}@allenai.org Abstract Large pre-trained models are capable of few- shot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input. However, the concate- nated demonstrations are often excessively long and induce additional computation. Inspired by fusion-in-decoder (FiD) models which effi- ciently aggregate more passages and thus out- performs concatenation-based models in open- domain QA, we hypothesize that similar tech- niques can be applied to improve the efficiency and end-task performance of ICL. To verify this, we present a comprehensive study on ap- plying three fusion methods—concatenation- based (early fusion), FiD (intermediate), and ensemble-based (late)—to ICL. We adopt a meta-learning setup where a model is first trained to perform ICL on a mixture of tasks us- ing one selected fusion method, then evaluated on held-out tasks for ICL. Results on 11 held- out tasks show that FiD-ICL matches or outper- forms the other two fusion methods. Addition- ally, we show that FiD-ICL (1) is 10x faster at inference time compared to concat-based and ensemble-based ICL, as we can easily pre- compute the representations of in-context ex- amples and reuse them; (2) enables scaling up to meta-training 3B-sized models, which would fail for concat-based ICL.1 1 Introduction Large pre-trained models demonstrated remarkable performance in learning new language tasks via few-shot fine-tuning (FT)—initializing a model with pre-trained weights and optimizing it based on a few examples (Zhang et al., 2021). FT-based approaches currently achieve state-of-the-art per- formance (Liu et al., 2022), yet they require back- propagating and computing gradients over the full models, which can be prohibitive under memory and resource constraints. †Work done while interning at Allen Institute for AI. 1Code: https://github.com/INK-USC/FiD-ICL Concat-based ICL Fusion-in-decoder Ensemble-based ICL Example 1 Example 2 Example 3 Input Encoder Output Example 1 Example 2 Example 3 Input Encoder Decoder Output Encoder Encoder Encoder Example 1 Input Encoder Output Encoder Example 2 Input Output Final Output Decoder Decoder Decoder Concat. Raw Text "Early Fusion" Concat. Hidden Repr. "Intermediate Fusion" Aggregate Scores for Rank Classification "Late Fusion" Figure 1: Overview. In this study we compare different methods to incorporate examples for in-context learning. We term these as “fusion methods”. ⊕ marks where and how fusion is implemented. An alternative approach to few-shot learning is in-context learning (ICL). By concatenating a few examples and prepending them before the test in- stance, the model can perform a new task read- ily (Brown et al., 2020). ICL is more efficient at its “learning” stage, as it only uses one forward- pass and does not require gradients at all. Yet it is less efficient at inference stage, as the concate- nated examples can become overly long and induce …

Content truncated to 500 words