MaChAmp at SemEval-2022 Tasks 2, 3, 4, 6, 10, 11, and 12: Multi-task Multi-lingual Learning for a Pre-selected Set of Semantic Datasets

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1695 - 1703 July 14-15, 2022 ©2022 Association for Computational Linguistics MaChAmp at SemEval-2022 Tasks 2, 3, 4, 6, 10, 11, and 12: Multi-task Multi-lingual Learning for a Pre-selected Set of Semantic Datasets Rob van der Goot IT University of Copenhagen robv@itu.dk Abstract Previous work on multi-task learning in Natu- ral Language Processing (NLP) often incorpo- rated carefully selected tasks as well as care- fully tuning of architectures to share informa- tion across tasks. Recently, it has shown that for autoregressive language models, a multi- task second pre-training step on a wide variety of NLP tasks leads to a set of parameters that more easily adapt for other NLP tasks. In this paper, we examine whether a similar setup can be used in autoencoder language models using a restricted set of semantically oriented NLP tasks, namely all SemEval 2022 tasks that are annotated at the word, sentence or paragraph level. We first evaluate a multi-task model trained on all SemEval 2022 tasks that contain annotation on the word, sentence or paragraph level (7 tasks, 11 sub-tasks), and then evalu- ate whether re-finetuning the resulting model for each task specificially leads to further im- provements. Our results show that our mono- task baseline, our multi-task model and our re- finetuned multi-task model each outperform the other models for a subset of the tasks. Overall, huge gains can be observed by doing multi-task learning: for three tasks we observe an error reduction of more than 40%.1 1 Introduction Recently, language models have become the de- facto standard in Natural Language Processing (NLP), where we first train a set of parameters on raw data, which are then finetuned on the task at hand. This in itself is a multi-task setup (language- modeling + target task). However, traditionally, multi-task learning was mainly done between mul- tiple NLP tasks with gold annotation. In this setup, many questions arise: not only how to share the in- formation between different tasks, but also when to share and even which tasks to use, as it is non-trivial 1code available at: https://bitbucket.org/ robvanderg/semeval2022 to decide which auxiliary tasks are beneficial for a certain target task (Ruder, 2017; Crawshaw, 2020). Early work on multi-task learning in NLP often used up to a handful of tasks, carefully curated dataset/task combinations, and carefully tuned how to share the information between these tasks (e.g. Hashimoto et al. (2017); Søgaard and Goldberg (2016)). A line of recent work has shown that an inter- mediate step can be used to finetune the language model on a set of NLP tasks, which leads to a model that is more apt for learning other NLP tasks. This is also called Supplementary Training on In- termediate Labeled-data Tasks (STILT), and was introduced by Phang et al. (2018). Phang et al. (2018) train on three classification tasks from the GLUE benchmark (Wang et al., 2018), and then retrain for all GLUE tasks, showing a 1.4 point of improvement …

Content truncated to 500 words