Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3016–3022 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3016 Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations Vladimir Araujo1,2, Andrés Villa1, Marcelo Mendoza3, Marie-Francine Moens2, Alvaro Soto1 1Pontificia Universidad Católica de Chile, 2KU Leuven, 3Universidad Técnica Federico Santa María {vgaraujo,afvilla}@uc.cl, mmendoza@inf.utfsm.cl sien.moens@kuleuven.be, asoto@ing.puc.cl Abstract Current language models are usually trained using a self-supervised scheme, where the main focus is learning representations at the word or sentence level. However, there has been limited progress in generating useful discourse-level representations. In this work, we propose to use ideas from predictive coding theory to augment BERT-style language mod- els with a mechanism that allows them to learn suitable discourse-level representations. As a result, our proposed approach is able to predict future sentences using explicit top-down con- nections that operate at the intermediate layers of the network. By experimenting with bench- marks designed to evaluate discourse-related knowledge using pre-trained sentence repre- sentations, we demonstrate that our approach improves performance in 6 out of 11 tasks by excelling in discourse relationship detection. 1 Introduction Pre-trained language models are among the leading methods to learn useful representations for textual data. Several pre-training objectives have been pro- posed in recent years, such as causal language mod- eling (Radford et al., 2018, 2019), masked language modeling (Devlin et al., 2019), and permutation language modeling (Yang et al., 2019). However, these approaches do not produce suitable represen- tations at the discourse level (Huber et al., 2020). Simultaneously, neuroscience studies have sug- gested that predictive coding (PC) plays an es- sential role in language development in humans (Ylinen et al., 2016; Zettersten, 2019). PC postu- lates that the brain is continually making predic- tions of incoming sensory stimuli (Rao and Ballard, 1999; Friston, 2005; Clark, 2013; Hohwy, 2013), with word prediction being the main mechanism (Berkum et al., 2005; Kuperberg and Jaeger, 2015). However, recent studies speculate that the predic- tive process could occur within and across utter- ances, fostering discourse comprehension (Kandy- laki et al., 2016; Pickering and Gambi, 2018). In this work, we propose to extend BERT-type models with recursive bottom-up and top-down computation based on PC theory. Specifically, we incorporate top-down connections that, according to PC, convey predictions from upper to lower lay- ers, which are contrasted with bottom-up represen- tations to generate an error signal that is used to guide the optimization of the model. Using this approach, we attempt to build feature representa- tions that capture discourse-level relationships by continually predicting future sentences in a latent space. We evaluate our approach on DiscoEval (Chen et al., 2019) and SciDTB for discourse eval- uation (Huber et al., 2020) to assess whether the embeddings produced by our model capture dis- course properties of sentences without finetuning. Our model achieves competitive performance com- pared to baselines, especially in tasks that require to discover discourse-level relations. 2 Related Work 2.1 BERT for Sentence Representation Pre-trained self-supervised language models have become popular in recent …

Content truncated to 500 words