Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9647–9660 July 9-14, 2023 ©2023 Association for Computational Linguistics Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answering Hai Ye Qizhe Xie Hwee Tou Ng Department of Computer Science, National University of Singapore {yehai,qizhex,nght}@comp.nus.edu.sg Abstract In this work, we study multi-source test-time model adaptation from user feedback, where K distinct models are established for adaptation. To allow efficient adaptation, we cast the prob- lem as a stochastic decision-making process, aiming to determine the best adapted model after adaptation. We discuss two frameworks: multi-armed bandit learning and multi-armed dueling bandits. Compared to multi-armed ban- dit learning, the dueling framework allows pair- wise collaboration among K models, which is solved by a novel method named Co-UCB pro- posed in this work. Experiments on six datasets of extractive question answering (QA) show that the dueling framework using Co-UCB is more effective than other strong baselines for our studied problem1. 1 Introduction Large language models (LLMs) can be fine-tuned or prompted with texts to achieve good perfor- mance in NLP tasks (Devlin et al., 2019; Brown et al., 2020; Ouyang et al., 2022). However, be- cause of the unexpected distribution shift at test time, the effectiveness of LLMs can degenerate (Wang et al., 2021c). They may also generate out- puts that are untrustworthy or toxic and fail to meet user expectations (Ouyang et al., 2022). One criti- cal issue that we need to address is to improve the generalization ability of LLMs. Recent research on test-time adaptation (TTA) suggests a possible way to do this, by continually updating the deployed model with target data from an arbitrary test distri- bution (Wang et al., 2021a). Interacting with users is important during test- time adaptation. First, user feedback allows the model to better align with humans (Stiennon et al., 2020; Ouyang et al., 2022). Users can directly teach the model to learn by interaction so that the 1Code of the paper is available at https://github.com/ oceanypt/Multi-source-TTA. 𝑆! 𝑆" 𝑆# user query model response user feedback model update source models User go over time Figure 1: The illustration of multi-source test-time adaptation from user feedback studied in this work. Each model is trained from a distinct source do- main. With unlabeled test data, models are adapted online from user feedback. model can be better trained to follow human in- structions and reduce the generation of toxic and harmful content. Besides, obtaining feedback from users can also reduce the cost of data annotation by experts, and the collected data will be more in line with the distribution of the users (Nguyen et al., 2017; Gao et al., 2022), which makes the adaptation more economical and effective. Leveraging multiple learned models of tasks is also important for TTA. As in previous work, uti- lizing multiple known tasks helps the model better learn new tasks (or distributions), such as meta- learning (Hospedales et al., 2022) and multi-source domain adaptation (Ramponi and Plank, …

Content truncated to 500 words