When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7465–7488 July 9-14, 2023 ©2023 Association for Computational Linguistics When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP Jingwei Ni ETH Zürich njingwei@ethz.ch Zhijing Jin MPI & ETH Zürich jinzhi@ethz.ch Qian Wang University of Zürich qian.wang@uzh.ch Mrinmaya Sachan ETH Zürich msachan@ethz.ch Markus Leippold University of Zürich & SFI markus.leippold@bf.uzh.ch Abstract Multi-task learning (MTL) aims at achieving a better model by leveraging data and knowl- edge from multiple tasks. However, MTL does not always work – sometimes negative transfer occurs between tasks, especially when aggre- gating loosely related skills, leaving it an open question when MTL works. Previous studies show that MTL performance can be improved by algorithmic tricks. However, what tasks and skills should be included is less well explored. In this work, we conduct a case study in Fi- nancial NLP where multiple datasets exist for skills relevant to the domain, such as numeric reasoning and sentiment analysis. Due to the task difficulty and data scarcity in the Finan- cial NLP domain, we explore when aggregat- ing such diverse skills from multiple datasets with MTL can work. Our findings suggest that the key to MTL success lies in skill diversity, relatedness between tasks, and choice of aggre- gation size and shared capacity. Specifically, MTL works well when tasks are diverse but related, and when the size of the task aggrega- tion and the shared capacity of the model are balanced to avoid overwhelming certain tasks.1 1 Introduction Multi-task learning (MTL) is a machine learn- ing paradigm where multiple learning tasks are optimized simultaneously, exploiting commonali- ties and differences across them (Caruana, 1997). MTL is expected to outperform single-task learning (STL) as it utilizes more training data and enables inter-task knowledge sharing (Ruder, 2017). How- ever, MTL may also bring about multi-task conflict and negative transfer. Empirically, in many MTL systems, only a small portion of tasks benefit from MT joint training while others suffer from negative transfer (Stickland and Murray, 2019; Raffel et al., 2020; Peng et al., 2020). Therefore, it is still an open question when MTL will work. 1https://github.com/EdisonNi-hku/MTL4Finance. Method TSA↓ SC↑ GPT-3 Zero-Shot 0.3700 77.69% GPT-3 Few-Shot 0.3128 80.37% FinBERT Fine-Tune 0.2054 86.61% Table 1: Performance comparison of our method (Fin- BERT Fine-Tune) and GPT-3 (text-davinci-003) base- lines. We report the rooted mean square error (↓) on the task of target-based sentiment analysis (TSA) (Cortis et al., 2017) and accuracy (↑) on sentiment classifica- tion (SC) (Malo et al., 2013). See GPT-3 prompts and settings in Appendix B. MTL systems have two components: MTL al- gorithms and the tasks included for aggregation. Recent progress in MTL has shown that appropri- ate MTL algorithms (e.g., architecture and opti- mization) can mitigate negative transfers (Yu et al., 2020; Wang et al., 2021; Pfeiffer et al., 2021; Karimi Mahabadi et al., 2021; Mao et al., 2022; Ponti et al., 2023, inter alia). However, it is still …

Content truncated to 500 words