CogTaskonomy: Cognitively Inspired Task Taxonomy Is Beneficial to Transfer Learning in NLP

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 904 - 920 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CogTaskonomy: Cognitively Inspired Task Taxonomy Is Beneficial to Transfer Learning in NLP Yifei Luo† , Minghui Xu† and Deyi Xiong∗ College of Intelligence and Computing, Tianjin University, Tianjin, China {yfluo,xuminghui,dyxiong}@tju.edu.cn Abstract Is there a principle to guide transfer learning across tasks in natural language processing (NLP)? Taxonomy (Zamir et al., 2018) finds that a structure exists among visual tasks, as a principle underlying transfer learning for them. In this paper, we propose a cognitively inspired framework, CogTaskonomy, to learn taxonomy for NLP tasks. The framework consists of Cognitive Representation Analytics (CRA) and Cognitive-Neural Mapping (CNM). The former employs Representational Similarity Analysis, which is commonly used in computational neu- roscience to find a correlation between brain- activity measurement and computational mod- eling, to estimate task similarity with task- specific sentence representations. The latter learns to detect task relations by projecting neu- ral representations from NLP models to cogni- tive signals (i.e., fMRI voxels). Experiments on 12 NLP tasks, where BERT/TinyBERT are used as the underlying models for transfer learning, demonstrate that the proposed Cog- Taskonomy is able to guide transfer learning, achieving performance competitive to the An- alytic Hierarchy Process (Saaty, 1987) used in visual Taskonomy (Zamir et al., 2018) but without requiring exhaustive pairwise O(m2) task transferring. Analyses further discover that CNM is capable of learning model- agnostic task taxonomy. The source code is available at https://github.com/ tjunlp-lab/CogTaskonomy.git. 1 Introduction Transfer learning (TL) has attracted extensive re- search interests in natural language processing with a wide range of forms, e.g., TL from pretrained language models (PLM) to downstream tasks (De- vlin et al., 2018; Radford et al., 2018), from a task with rich labeled data to a task with low resource †Equal contribution. *Corresponding author. (Chu and Wang, 2018; Yu et al., 2021), from high- resource languages to low-resource languages (Gu et al., 2018; Ko et al., 2021), etc.1 A high-level concept or question on cross-task transfer learn- ing is how these involved tasks are related to each other. Is sentiment analysis related to paraphras- ing? Is textual entailment more related to question answering than named entity recognition? All these sub-questions resolve themselves into whether a structure exists among NLP tasks. Such task taxon- omy is of notable values to transfer learning in NLP in that it has the potential to guide TL and reduce redundancies across tasks (Zamir et al., 2018). In this paper, partially inspired by the task taxon- omy in visual tasks (Zamir et al., 2018), we study the hierarchical task structure for NLP tasks. But significantly different from the visual Taskonomy (Zamir et al., 2018), we construct NLP taskonomy from a cognitively inspired perspective. Cognitively inspired NLP is the intersection of NLP and cognitive neuroscience that aims at un- covering cognitive processes in the brain, including cognition in language comprehension. With the in- creasing availability of cognitively annotated …

Content truncated to 500 words