Towards Zero-shot Language Modeling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2900–2910 Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics ACL Anthology [revision] 6 Aug 2021 Towards Zero-shot Language Modeling Edoardo M. Ponti1, Ivan Vuli´c1, Ryan Cotterell2, Roi Reichart3, Anna Korhonen1 1Language Technology Lab, TAL, University of Cambridge 2Computer Laboratory, University of Cambridge 2Faculty of Industrial Engineering and Management, Technion, IIT 1,2{ep490,iv250,rdc42,alk23}@cam.ac.uk 3roiri@ie.technion.ac.il Abstract Can we construct a neural model that is in- ductively biased towards learning human lan- guages? Motivated by this question, we aim at constructing an informative prior over neu- ral weights, in order to adapt quickly to held- out languages in the task of character-level lan- guage modeling. We infer this distribution from a sample of typologically diverse train- ing languages via Laplace approximation. The use of such a prior outperforms baseline mod- els with an uninformative prior (so-called ‘fine- tuning’) in both zero-shot and few-shot set- tings. This shows that the prior is imbued with universal phonological knowledge. More- over, we harness additional language-specific side information as distant supervision for held-out languages. Specifically, we condition language models on features from typologi- cal databases, by concatenating them to hid- den states or generating weights with hyper- networks. These features appear beneficial in the few-shot setting, but not in the zero-shot setting. Since the paucity of digital texts af- fects the majority of the world’s languages, we hope that these findings will help broaden the scope of applications for language technology. 1 Introduction With the success of recurrent neural networks and other black-box models on core NLP tasks, such as language modeling, researchers have turned their attention to the study of the inductive bias such neu- ral models exhibit (Linzen et al., 2016; Marvin and Linzen, 2018; Ravfogel et al., 2018). A number of natural questions have been asked. For example, do recurrent neural language models learn syntax (Marvin and Linzen, 2018)? Do they map onto grammaticality judgments (Warstadt et al., 2019)? However, as Ravfogel et al. (2019) note, “[m]ost of the work so far has focused on English.” Moreover, these studies have almost always focused on train- ing scenarios where a large number of in-language sentences are available. In this work, we aim to find a prior distribu- tion over network parameters that generalize well to new human languages. The recent vein of re- search on the inductive biases of neural nets im- plicitly assumes a uniform (unnormalizable) prior over the space of neural network parameters (Rav- fogel et al., 2019, inter alia). In contrast, we take a Bayesian-updating approach: First, we approxi- mate the posterior distribution over the network pa- rameters using the Laplace method (Azevedo-Filho and Shachter, 1994), given the data from a sample of seen training languages. Afterward, this distri- bution serves as a prior for maximum-a-posteriori (MAP) estimation of network parameters for the held-out unseen languages. The search for a universal prior for linguistic knowledge is motivated …

Content truncated to 500 words