Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2971–2985 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2971 Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks Fynn Schr¨oder Language Technology Group Universit¨at Hamburg Hamburg, Germany fschroeder@informatik.uni-hamburg.de Chris Biemann Language Technology Group Universit¨at Hamburg Hamburg, Germany biemann@informatik.uni-hamburg.de Abstract Multi-task learning (MTL) and transfer learn- ing (TL) are techniques to overcome the is- sue of data scarcity when training state-of-the- art neural networks. However, finding bene- ficial auxiliary datasets for MTL or TL is a time- and resource-consuming trial-and-error approach. We propose new methods to au- tomatically assess the similarity of sequence tagging datasets to identify beneficial auxiliary data for MTL or TL setups. Our methods can compute the similarity between any two se- quence tagging datasets, i.e. they do not need to be annotated with the same tagset or multi- ple labels in parallel. Additionally, our meth- ods take tokens and their labels into account, which is more robust than only using either of them as an information source, as conducted in prior work. We empirically show that our sim- ilarity measures correlate with the change in test score of neural networks that use the auxil- iary dataset for MTL to increase the main task performance. We provide an efficient, open- source implementation.1 1 Introduction State-of-the-art neural networks usually require large amounts of training data and vast compu- tational resources. Especially for low-resource tasks, data scarcity is the main issue hampering the training of robust models. By leveraging multi- task learning or transfer learning, auxiliary data can be incorporated into the training to boost the main task performance. Finding suitable auxiliary datasets for these cases is a time- and resource- consuming trial-and-error approach, because there can be plenty of plausible auxiliary datasets that could help to learn the main task. For a proper evaluation of different auxiliary datasets, hyperpa- rameter search and training runs with multiple ran- dom seeds have to be performed for each auxiliary 1github.com/uhh-lt/seq-tag-sim dataset individually. Thus, the process takes even longer and uses even more computational resources. We propose methods to shorten this trial-and-error approach by computing the similarity between any two sequence tagging datasets. Based on the simi- larity, suitable datasets can be quickly selected to be used as auxiliary training data for multi-task or transfer learning. Our contributions are a family of novel meth- ods to compute the similarity of sequence tag- ging datasets, where the similarity values correlate with the change in multi-task learning performance when using one dataset as auxiliary data for train- ing the other. We evaluate our methods in experi- ments with five part-of-speech (POS) tagging, nine named-entity recognition (NER) and three argu- mentation mining (AM) datasets. Our similarity measures allow for comparison both datasets for the same and different tasks, not requiring the same set of labels on target and auxiliary dataset. The calculated similarity scores can be used to predict which dataset will be beneficial as auxiliary …

Content truncated to 500 words