Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages Edoardo M. Pontiκ, Ivan Vuli´cκ, Ryan Cotterellκ, ζ, Marinela Parovi´cκ, Roi Reichartτ, Anna Korhonenκ κUniversity of Cambridge ζETH Z¨urich τTechnion, IIT κ{ep490,iv250,rdc42,mp939,alk23}@cam.ac.uk τroiri@ie.technion.ac.il Abstract Most combinations of NLP tasks and lan- guage varieties lack in-domain examples for supervised training because of the paucity of annotated data. How can neural models make sample-efficient generalizations from task–language combinations with available data to low-resource ones? In this work, we propose a Bayesian generative model for the space of neural parameters. We assume that this space can be factorized into latent vari- ables for each language and each task. We infer the posteriors over such latent variables based on data from seen task–language combinations through variational inference. This enables zero-shot classification on unseen combina- tions at prediction time. For instance, given training data for named entity recognition (NER) in Vietnamese and for part-of-speech (POS) tagging in Wolof, our model can per- form accurate predictions for NER in Wolof. In particular, we experiment with a typolog- ically diverse sample of 33 languages from 4 continents and 11 families, and show that our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods. Our code is available at github.com/cambridgeltl/parameter -factorization. 1 Introduction The annotation efforts in NLP have achieved im- pressive feats, such as the Universal Dependen- cies (UD) project (Nivre et al., 2019), which now includes 83 languages. But even UD covers only a meager subset of the world’s estimated 8,506 languages (Hammarström et al., 2016). What is more, the Association for Computational Linguis- tics Wiki1 lists 24 separate NLP tasks. Labeled data, which is both costly and labor-intensive, is missing for many of such task–language combi- nations. This shortage hinders the development of computational models for the majority of the world’s languages (Snyder and Barzilay, 2010; Ponti et al., 2019a). A common solution is transferring knowledge across domains, such as tasks and languages (Yogatama et al., 2019; Talmor and Berant, 2019), which holds promise to mitigate the lack of training data inherent to a large spectrum of NLP applications (T¨ackstr¨om et al., 2012; Agi´c et al., 2016; Ammar et al., 2016; Ponti et al., 2018; Ziser and Reichart, 2018, inter alia). In the most extreme scenario, zero-shotlearning, no annotated examples are available for the target domain. In particular, zero-shot transfer across languages implies a change in the data domain, and leverages information from resource-rich languages to tackle the same task in a previously unseen target language (Lin et al., 2019; Rijhwani et al., 2019; Artetxe and Schwenk, 2019; Ponti et al., 2019a, inter alia). Zero-shot transfer across tasks within the same language (Ruder et al., 2019a), on the other hand, implies a change in the space of labels. As our main contribution, we propose a Bayesian generative model of the neural parame- ter space. We assume this to be structured, and for this reason factorizable into task- and language- specific latent variables.2 By performing transfer of knowledge from both related tasks and …

Content truncated to 500 words