FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10936 - 10953 December 7-11, 2022 ©2022 Association for Computational Linguistics FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue Alon Albalak1 Yi-Lin Tuan1 Pegah Jandaghi2 Connor Pryor3 Luke Yoffe1 Deepak Ramachandran4 Lise Getoor3 Jay Pujara2 William Yang Wang1 1University of California, Santa Barbara 2University of Southern California 3University of California, Santa Cruz 4Google Research alon_albalak@ucsb.edu Abstract Task transfer, transferring knowledge contained in related tasks, holds the promise of reducing the quantity of labeled data required to fine- tune language models. Dialogue understand- ing encompasses many diverse tasks, yet task transfer has not been thoroughly studied in con- versational AI. This work explores conversa- tional task transfer by introducing FETA: a benchmark for FEw-sample TAsk transfer in open-domain dialogue. FETA contains two underlying sets of conversations upon which there are 10 and 7 tasks annotated, enabling the study of intra-dataset task transfer; task transfer without domain adaptation. We uti- lize three popular language models and three learning algorithms to analyze the transferabil- ity between 132 source-target task pairs and create a baseline for future work. We run exper- iments in the single- and multi-source settings and report valuable findings, e.g., most per- formance trends are model-specific, and span extraction and multiple-choice tasks benefit the most from task transfer. In addition to task transfer, FETA can be a valuable resource for future research into the efficiency and gener- alizability of pre-training datasets and model architectures, as well as for learning settings such as continual and multitask learning. 1 1 Introduction Improving sample efficiency through transfer learn- ing has been a long-standing challenge in the ma- chine learning and natural language processing communities (Pratt et al., 1991; Ando and Zhang, 2005). Dialogue data requires multiple cohesive turns with consistent speaker personalities (Ur- banek et al., 2019; Huang et al., 2020), creating a challenge for data collection and motivating the development of techniques that improve sample efficiency in conversational AI (Lin et al., 2020). 1Benchmark available at alon-albalak.github.io/feta- website. We utilize the Transfer Learning in Dialogue Bench- marking Toolkit for all experiments (TLiDB python package). Figure 1: Task Transfer Performance on FETA- DailyDialog. Computed transfer performance is demon- strated by arrows leaving from source tasks and entering target tasks. Strength of the transfer is denoted by thick- ness and color of edges. Furthermore, dialogue understanding tasks re- quire a shared knowledge of semantics, pragmatics, human behavior, and commonsense, making dia- logue an area of study that can benefit greatly from a deeper understanding of transfer learning. Two essential transfer learning settings, namely domain adaptation and task transfer, have been studied on language tasks (Ruder et al., 2019). While domain adaptation has been studied in task- oriented dialogue (Mehri et al., 2020) , task transfer has been studied with less rigor in conversational AI. Prior studies of task transfer in dialogue con- sider only 2-4 tasks, focus on multitask learning, and do not compare learning algorithms (Hosseini- Asl et al., …

Content truncated to 500 words