MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets.

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 230–245 July 13-14, 2023 ©2023 Association for Computational Linguistics MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets. Rob van der Goot IT University of Copenhagen robv@itu.dk Abstract To improve the ability of language models to handle Natural Language Processing (NLP) tasks and intermediate step of pre-training has recently been introduced. In this setup, one takes a pre-trained language model, trains it on a (set of) NLP dataset(s), and then finetunes it for a target task. It is known that the selec- tion of relevant transfer tasks is important, but recently some work has shown substantial per- formance gains by doing intermediate training on a very large set of datasets. Most previous work uses generative language models or only focuses on one or a couple of tasks and uses a carefully curated setup. We compare inter- mediate training with one or many tasks in a setup where the choice of datasets is more arbit- rary; we use all SemEval 2023 text-based tasks. We reach performance improvements for most tasks when using intermediate training. Gains are higher when doing intermediate training on single tasks than all tasks if the right trans- fer task is identified. Dataset smoothing and heterogeneous batching did not lead to robust gains in our setup. 1 1 Introduction The introduction of word embeddings and later con- textualized transformer-based models, i.e. Large Language Models (LLM), have led to perform- ance improvements on many Natural Language Processing tasks. Technically these approaches are multi-task learning approaches (with parameter sharing over time), where we first train a language model on raw data, save the weights, and then re- train the weights for our target task. It has been shown that intermediate steps of training can be beneficial for downstream performance. This in- termediate step of training can be done through language modeling to adapt to new domains or languages (Gururangan et al., 2020; Muller et al., 1Code available at: https://bitbucket.org/ robvanderg/semeval2023/ 2021) or on NLP datasets directly (Phang et al., 2018; Aribandi et al., 2022), exploiting the syner- gies across tasks. Intermediate training on NLP datasets has benefits both for performance and effi- ciency (retraining on tasks is cheaper compared to training a full language model). One important desideratum in finding the right language model for a downstream task is the train- ing data and its distance to the target data. For an intermediate training step on NLP datasets this is even more complex, as there are more design decisions to make; e.g.: which NLP tasks are relev- ant for the target task? Which datasets are closest to the target data? How many (or how much) of them do we use? Which LLM to use as a start- ing point? Previous work has investigated mostly single dimensions in this choice of intermediate training datasets (in a carefully curated setup), less is …

Content truncated to 500 words