Probing BERT’s priors with serial reproduction chains

Findings of the Association for Computational Linguistics: ACL 2022, pages 3977 - 3992 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Probing BERT’s priors with serial reproduction chains Takateru Yamakoshi1,2, Thomas L. Griffiths1, Robert D. Hawkins1 1Princeton University, 2The University of Tokyo {takateru,tomg,rdhawkins}@princeton.edu Abstract Sampling is a promising bottom-up method for exposing what generative models have learned about language, but it remains unclear how to generate representative samples from popular masked language models (MLMs) like BERT. The MLM objective yields a dependency net- work with no guarantee of consistent condi- tional distributions, posing a problem for naive approaches. Drawing from theories of iter- ated learning in cognitive science, we explore the use of serial reproduction chains to sam- ple from BERT’s priors. In particular, we ob- serve that a unique and consistent estimator of the ground-truth joint distribution is given by a Generative Stochastic Network (GSN) sam- pler, which randomly selects which token to mask and reconstruct on each step. We show that the lexical and syntactic statistics of sen- tences from GSN chains closely match the ground-truth corpus distribution and perform better than other methods in a large corpus of naturalness judgments. Our findings establish a firmer theoretical foundation for bottom-up probing and highlight richer deviations from human priors1. 1 Introduction Large neural language models have become the representational backbone of natural language pro- cessing. By learning to predict words from their context, these models have induced surprisingly human-like linguistic knowledge, from syntactic structure (Linzen and Baroni, 2021; Tenney et al., 2019; Warstadt et al., 2019) and subtle lexical bi- ases (Hawkins et al., 2020) to more insidious so- cial biases and stereotypes (Caliskan et al., 2017; Garg et al., 2018). At the same time, efforts to probe these models have revealed significant de- viations from natural language (Braverman et al., 2020; Holtzman et al., 2019; Dasgupta et al., 2020). 1Code and data are available at https://github. com/taka-yamakoshi/TelephoneGame chain 2 chain 1 food was running short, and winters were colder. time was running short, and winters were colder. time was running out, and winters were colder. Figure 1: We use a serial reproduction method to probe BERT’s prior over possible sentences (visualization of reproduction chains obtained by running t-sne on sen- tence embeddings; chains are color-coded and fade to black across their burn-in period). Observations of incoherent or “weird” behavior may often be amusing, as when a generated recipe begins with “1/4 pounds of bones or fresh bread” (Shane, 2019), but also pose significant dangers in real-world settings (Bender et al., 2021). These deviations present a core theoretical and methodological puzzle for computational linguis- tics. How do we elicit and characterize the full prior2 that a particular model has learned over pos- sible sentences in a language? A dominant ap- proach has been to design benchmark suites that 2We use the term prior to refer to graded linguistic knowl- edge assigning probabilities to all possible sentences. While we focus on text, this prior is also the foundation for more …

Content truncated to 500 words