A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 943–956 May 2-6, 2023 ©2023 Association for Computational Linguistics A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, Meng Jiang University of Notre Dame, Notre Dame, IN, USA {zzhang23, wyu1, myu2, zguo5, mjiang2}@nd.edu Abstract Multi-task learning (MTL) has become increas- ingly popular in natural language processing (NLP) because it improves the performance of related tasks by exploiting their commonalities and differences. Nevertheless, it is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks. In this survey, we review re- cent advances of multi-task learning methods in NLP, with the aim of summarizing them into two general multi-task training methods based on their task relatedness: (i) joint training and (ii) multi-step training. We present examples in various NLP downstream applications, sum- marize the task relationships and discuss future directions of this promising topic. 1 Introduction Machine learning generally involves training a model to perform a single task. By focusing on one task, the model ignores knowledge from the train- ing signals of related tasks (Ruder, 2017). There are a great number of tasks in NLP, from syntax parsing to information extraction, from machine translation to question answering: each requires a model dedicated to learning from data. Biologi- cally, humans learn natural languages, from basic grammar to complex semantics in a single brain (Hashimoto et al., 2017). In the field of machine learning, multi-task learning (MTL) aims to lever- age useful information shared across multiple re- lated tasks to improve the generalization perfor- mance on all tasks (Caruana, 1997). In deep neural networks, it is generally achieved by sharing part of hidden layers between different tasks, while keep- ing several task-specific output layers. MTL offers advantages like improved data efficiency, reduced overfitting, and fast learning by leveraging auxil- iary information (Crawshaw, 2020). There have been relevant surveys that looked into architecture designs and optimization algo- ������� ����������� ������ ������ ������� ����������� ��������� ������ D1 E1 D2 E2 Dn En … … ���� ���� ���� L1 L2 Ln T1 T2 Tn D1 E1 D2 E2 Dn En … … L1 L2 Ln T1 T2 Tn ��� ����� �������� ��� ���������� �������� ����������� ������� ��������� ������� Figure 1: Two multi-task learning frameworks. rithms in MTL. Ruder (2017) classified different MTL frameworks into two categories: hard param- eter sharing and soft parameter sharing, as well as some earlier MTL examples in both non-neural and neural models; Zhang and Yang (2018) ex- panded such two “how to share” categories into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach, and decomposition ap- proach; Crawshaw (2020) presented more recent models in both single-domain and multi-modal ar- chitectures, as well as an overview of optimization methods in MTL. Nevertheless, it is still not clearly understood how to design and train a single model …

Content truncated to 500 words