RELATE: Generating a linguistically inspired Knowledge Graph for fine-grained emotion classification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 633–647 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 633 Analysis and Prediction of NLP models via Task Embeddings Damien Sileo, Marie-Francine Moens KU Leuven Belgium damien.sileo@kuleuven.be, sien.moens@cs.kuleuven.be Abstract Task embeddings are low-dimensional representations that are trained to capture task properties. In this paper, we propose MetaEval, a collection of 101 NLP tasks. We fit a single transformer to all MetaEval tasks jointly while conditioning it on learned embeddings. The resulting task embeddings enable a novel analysis of the space of tasks. We then show that task aspects can be mapped to task embeddings for new tasks without using any annotated examples. Predicted embeddings can modulate the encoder for zero-shot inference and outperform a zero-shot baseline on GLUE tasks. The provided multitask setup can function as a benchmark for future transfer learning research. Keywords: task embeddings, metalearning, natural language processing, evaluation, extreme multi-task learning 1. Introduction Transfer between tasks enabled considerable progress in NLP. Pretrained transformer-based encoders, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), achieved state-of-the-art results on text clas- sification tasks. These models acquire rich text repre- sentations through masked language modeling (MLM) pretraining (Tenney et al., 2019; Warstadt et al., 2019; Warstadt et al., 2020b). However, these representations need additional task supervision to be useful for down- stream tasks (Reimers and Gurevych, 2019). The de- fault technique, full fine-tuning, optimizes all encoder weights alongside the training of the task-specific clas- sifier. The resulting encoder weights can be seen as a very high-dimensional1 continuous representation of a model that is dedicated to a task Ti (Aghajanyan et al., 2020). These weights provide a way to predict re- latedness between tasks (Achille et al., 2019; Vu et al., 2020). However, they are very high dimensional, and they cannot be modulated to adapt the network to unseen tasks. Hypernetworks (Ha et al., 2017; von Oswald et al., 2020; Hansen et al., 2020), i.e., neural networks whose weights are modulated by an outer net- work, can solve this problem. These techniques have been adapted to NLP by Pilault et al. (2021) and Ma- habadi et al. (2021) who rely on adapters (Houlsby et al., 2019). Adapters are parameter-efficient layers that can be inserted between specific layers and trained to modulate a frozen transformer. An adapter Ai is com- posed of distinct adapter layers. Pilault et al. (2021) and Mahabadi et al. (2021) showed that in a multitask setting with a collection of tasks Θ, a set of adapters {Ai, Ti ∈ Θ} can be decomposed into two components: a set of task embeddings {zi, Ti ∈ Θ} and a single shared conditional adapter A(zi). The task embeddings and conditional adapter are trained jointly, which allows each task to modulate the shared model in its own way. This approach leads to a performance improvement over individual adapters or full fine-tuning while allowing 1E.g., ≈ 110M dimensions for BERTBASE full …

Content truncated to 500 words