Language Modelling as a Multi-Task Problem

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2049–2060 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2049 Language Modelling as a Multi-Task Problem Lucas Weber DTCL, University Pompeu Fabra lucas.weber@upf.edu Jaap Jumelet ILLC, University of Amsterdam j.w.d.jumelet@uva.nl Elia Bruni IKW, University of Osnabr¨uck elia.bruni@gmail.com Dieuwke Hupkes Facebook AI Research dieuwkehupkes@fb.com Abstract In this paper, we propose to study language modelling as a multi-task problem, bring- ing together three strands of research: multi- task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language mod- els adhere to learning principles of multi-task learning during training. To showcase the idea, we analyse the generalisation behaviour of lan- guage models as they learn the linguistic con- cept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task set- ting naturally emerges within the objective of the more general task of language modelling. We argue that this insight is valuable for multi- task learning, linguistics and interpretability research and can lead to exciting new findings in all three domains. 1 Introduction Humans are optimising their behaviour towards a multitude of objectives to reach their goals in day- to-day life. By learning many things at the same time and exploiting their commonalities, they ac- quire more general knowledge about the world, which in turn helps them to learn new things quicker (Perkins et al., 1992; Schwartz et al., 2005; Cormier and Hagman, 2014; Luriia, 1976). This idea of finding more general solutions through the diversification of tasks has found its way also to the machine learning community, in the field of multi- task learning (MTL) (Caruana, 1993, 1997). In MTL, multiple tasks are optimised jointly, enabling the transfer of relevant information across tasks. MTL research yields fruitful results in both appli- cation (e.g. Collobert and Weston, 2008; Collobert et al., 2011; Zhang et al., 2014; Donahue et al., 2014; Kaiser et al., 2017) and theory (e.g. Bax- ter, 2000; Maurer, 2006; Ando and Zhang, 2005; Argyriou et al., 2008). However, deciding on a setup requires making many arbitrary choices. The researcher or engineer has to decide which tasks to train together (e.g. Bin- gel and Søgaard, 2017; Standley et al., 2020); at which hierarchy-level to allow tasks to interact (e.g. Søgaard and Goldberg, 2016); which degree of pa- rameter sharing to employ (Ruder, 2017); which distribution of training data to employ (e.g. Luong et al., 2016), and so on. Having to make so many arbitrary choices is inconvenient for modellers, but also stands in the way of understanding the learning principles of neural models in multi-task settings. The highly constructed learning scenarios make it difficult to see whether outcomes should be at- tributed to one of the many a-priori decisions or to inherent properties of the learning process. In this paper, we propose to study MTL not in a constructed, artificial scenario, but in a more nat- ural setting. To do so, we consider the objective of language …

Content truncated to 500 words