The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5555–5577 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5555 The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders Han He Department of Computer Science Emory University Atlanta, GA 30322 USA han.he@emory.edu Jinho D. Choi Department of Computer Science Emory University Atlanta, GA 30322 USA jinho.choi@emory.edu Abstract Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a ques- tion still remains whether or not it would per- form as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we pro- pose the Stem Cell Hypothesis to reveal the ex- istence of attention heads naturally talented for many tasks that cannot be jointly trained to cre- ate adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis. 1 Introduction Transformer encoders (TEs) have established re- cent state-of-the-art results on many core NLP tasks (He and Choi, 2019; Yu et al., 2020; Zhang et al., 2020). However, their architectures can be viewed “over-parameterized” as downstream tasks may not need all those parameters, prone to cause an over- head in computation. One promising approach to mitigate this overhead is multi-task learning (MTL) where a TE is shared across multiple tasks; thus, it needs to be run only once to generate final embed- dings for all tasks (Clark et al., 2019b). Despite the success in MTL on closely-related tasks such as language understanding (Wang et al., 2018) or relation extraction (Chen et al., 2020; Lin et al., 2020), MTL on core NLP tasks (e.g., tagging, parsing, labeling) whose decoders are very distinct has not been well-studied. This work employs the state-of-the-art decoders on five core tasks for MTL and thoroughly analyzes interactions among those tasks to explore a possibility of reducing the compu- tation overhead from TEs. Surprisingly, our exper- iments depict that models jointly trained by MTL give lower accuracy than ones trained individually, that is against findings from previous work. In fact, models jointly trained with all five tasks perform the worst among any other combination (Section 3). These experimental results urge us to figure out why MTL on core tasks with a shared TE leads to worse performance than its single-task counterparts. Our exploration begins by detecting essential heads for each task by forcing the TE to use as few atten- tion heads as possible while maintaining accuracy similar to a fully-utilized encoder. Our experiments reveal that …

Content truncated to 500 words