Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings
37644Authors
nan
Abstract
AbstractAdaptive inference is a simple method for reducing inference costs. The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty. In this work, we compare the two main approaches for adaptive inference, Early-Exit and Multi-Model, when training data is limited. First, we observe that for models with the same architecture and size, individual Multi-Model classifiers outperform their Early-Exit counterparts by an average of 2.3%. We show that this gap is caused by Early-Exit classifiers sharing model parameters during training, resulting in conflicting gradient updates of model weights. We find that despite this gap, Early-Exit still provides a better speed-accuracy trade-off due to the overhead of the Multi-Model approach. To address these issues, we propose SWEET (Separating Weights for Early-Exit Transformers) an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. We compare SWEET’s speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find that it outperforms both methods at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. Moreover, SWEET individual classifiers outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of both methods, paving the way for further reduction of inference costs in NLP.
Keywords
Full Content
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1471–1488 July 9-14, 2023 ©2023 Association for Computational Linguistics COMPOsitional Data Augmentation for Abstractive Conversation Summarization Siru Ouyang1, Jiaao Chen2, Jiawei Han1, Diyi Yang3 1 University of Illinois Urbana-Champaign 2 Georgia Institute of Technology 3 Stanford University {siruo2,hanj}@illinois.edu, jchen896@gatech.edu, diyiy@stanford.edu Abstract Recent abstractive conversation summarization systems generally rely on large-scale datasets with annotated summaries. However, collect- ing and annotating these conversations can be a time-consuming and labor-intensive task. To address this issue, in this work, we present a sub-structure level compositional data aug- mentation method, COMPO, for generating di- verse and high-quality pairs of conversations and summaries. Specifically, COMPO first ex- tracts conversation structures like topic splits and action triples as basic units. Then we or- ganize these semantically meaningful conver- sation snippets compositionally to create new training instances. Additionally, we explore noise-tolerant settings in both self-training and joint-training paradigms to make the most of these augmented samples. Our experiments on benchmark datasets, SAMSum and Di- alogSum, show that COMPO substantially out- performs prior baseline methods by achiev- ing a nearly 10% increase of ROUGE scores with limited data. We have publically re- leased our code at https://github.com/ ozyyshr/Compo. 1 Introduction Abstractive conversation summarization, which condenses unstructured conversations into short, concise, and structured text, has greatly benefited from neural generative models trained on large- scale annotated data. Researchers have focused on various aspects in conversation summarization, such as hierarchical modeling of conversations (Zhao et al., 2019; Zhu et al., 2020), leveraging dia- logue acts (Goo and Chen, 2018), using key phrases and entities (Liu et al., 2019a; Narayan et al., 2021), utilizing topic segments (Liu et al., 2019b), incor- porating stage components (Chen and Yang, 2020) and examining discourse relations (Chen and Yang, 2021b; Feng et al., 2020b). However, training these generative models often requires abundant high- Mary: Sorry, I didn't make it to your birthday party :( Nick: It's OK... Mary: I just got so distracted! I forgot it was yesterday! Nick: do tell! Mary: I met this guy... Nick: REALLY? I want details :D Mary: Yeah, his name is Kirk and he's an architect... Nick: OK, just your type then #file_gif# Mary: And we ended up spending the whole week together. Nick: A WEEK? Mary: Yeah... It's madness, I'll tell you more this evening. Are we still on? Nick: You bet we are! Conversation Actions Summary Mary didn't come to Nick's birthday party. She met an architect named Kirk. Mary and Nick will meet in the evening. Mary, didn’t make, party Mary, got distracted Mary, forgot Nick, want details He, is, architect We, end up, spend Spend, weekend Mary, will tell, Nick Mary, meet, guy Figure 1: An example of conversation, extracted actions and its paired summary sentences (randomly sampled from SAMSum). The corresponding summary consists of three sentences, each sentence relates to one snippet (illustrated by color). quality data, i.e., conversation and its paired sum- mary, which is usually time-consuming …
Content truncated to 500 words
Labeled Text
AbstractAdaptive inference is a simple method for reducing inference costs. The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty. In this work, we compare the two main approaches for adaptive inference, Early-Exit and Multi-Model, when training data is limited. First, we observe that for models with the same architecture and size, individual Multi-Model classifiers outperform their Early-Exit counterparts by an average of 2.3%. We show that this gap is caused by Early-Exit classifiers sharing model parameters during training, resulting in conflicting gradient updates of model weights. We find that despite this gap, Early-Exit still provides a better speed-accuracy trade-off due to the overhead of the Multi-Model approach. To address these issues, we propose SWEET (Separating Weights for Early-Exit Transformers) an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. We compare SWEET’s speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find that it outperforms both methods at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. Moreover, SWEET individual classifiers outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of both methods, paving the way for further reduction of inference costs in NLP.
Publication Details
- Paper ID:
- 37644
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan