Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7302 - 7315 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models Ryokan Ri and Yoshimasa Tsuruoka The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan {li0123,tsuruoka}@logos.t.u-tokyo.ac.jp Abstract We investigate what kind of structural knowl- edge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural prop- erties that mimic natural language, pretrain encoders on the data, and see how much per- formance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natu- ral language. A follow-up probing analysis in- dicates that its success in the transfer is related to the amount of encoded contextual informa- tion and what is transferred is the knowledge of position-aware context dependence of lan- guage. Our results provide insights into how neural network encoders process human lan- guages and the source of cross-lingual transfer- ability of recent multilingual language models. 1 Introduction Pretrained language models (Devlin et al., 2019; Yang et al., 2019; Raffel et al., 2020) have demon- strated strong empirical performance not only within a language but also across languages. Lan- guage models pretrained with a mix of monolingual corpora, such as multilingual BERT, exhibit a de- cent zero-shot cross-lingual transfer capability, i.e., a model fine-tuned in a single source language (L1) can solve the task in another language (L2) (Con- neau et al., 2020a; Xue et al., 2021). Surprisingly, the transfer happens without lexical overlaps be- tween L1 and L2 (Karthikeyan K and Roth, 2020; Conneau et al., 2020b) or even without joint pre- training (Artetxe et al., 2020): an encoder only pretrained on L1 can be transferred to L2 without any parameter updates. These results suggest that, whether the encoder is trained on single or multiple languages, it learns some transferable knowledge about language. Figure 1: Transfer from artificial language to natural language. The artificial language encodes some struc- tural properties (e.g., token distributions, dependency structures) and we study how the learning of such prop- erties can be transferred to natural language. However, the characteristics of such transferable knowledge are still underexplored. Recent stud- ies with the probing methodology (Hupkes and Zuidema, 2018; Conneau et al., 2018) have re- vealed that multilingual BERT captures language- independent linguistic structures such as universal dependency relations (Chi et al., 2020) and subject- hood (Papadimitriou et al., 2021), but it remains unknown whether learning such linguistic proper- ties actually contributes to the performance, and whether there exists more abstract knowledge trans- ferred across languages. In this study, we try to shed light on these ques- tions with the framework of the Test for Inductive Bias via Language Model Transfer (Papadimitriou and Jurafsky, 2020), focusing on designing arti- ficial languages with natural-language-like struc- tural properties (Figure 1). We pretrain encoders 7302 with …

Content truncated to 500 words