CogBERT: Cognition-Guided Pre-trained Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 3210–3225 October 12–17, 2022. 3210 CogBERT: Cognition-Guided Pre-trained Language Models Xiao Ding, Bowen Chen, Li Du, Bing Qin, Ting Liu Harbin Institute of Technology {xding, bwchen, ldu, qbing, tliu}@ir.hit.edu.cn Abstract We study the problem of integrating cogni- tive language processing signals (e.g., eye- tracking or EEG data) into pre-trained mod- els like BERT. Existing methods typically fine- tune pre-trained models on cognitive data, ig- noring the semantic gap between the texts and cognitive signals. To fill the gap, we propose CogBERT, a framework that can induce fine- grained cognitive features from cognitive data and incorporate cognitive features into BERT by adaptively adjusting the weight of cognitive features for different NLP tasks. Extensive ex- periments show that: (1) Cognition-guided pre- trained models can consistently perform better than basic pre-trained models on ten NLP tasks. (2) Different cognitive features contribute dif- ferently to different NLP tasks. Based on this observation, we give a fine-grained explana- tion of why cognitive data is helpful for NLP. (3) Different transformer layers of pre-trained models should encode different cognitive fea- tures, with word-level cognitive features at the bottom and semantic-level cognitive features at the top. (4) Attention visualization demon- strates that CogBERT can align with human gaze patterns and improves its natural language understanding ability.1 1 Introduction Pre-trained models such as BERT (Devlin et al., 2019), GPT (Radford and Narasimhan, 2018) and RoBERTa (Liu et al., 2019) have brought promis- ing improvements to natural language processing (NLP) tasks, such as event prediction (Li et al., 2019) and sentiment analysis (Hoang et al., 2019). On the other hand, from a language processing perspective, cognitive neuroscience studies the bi- ological and cognitive processes underlying lan- guage processing in human brains. Researchers specifically designed pre-trained models to cap- ture how the brain represents language meaning 1Our code will be released upon publication. The ISIS has done so many terrible things in Syria and caused chaos in the Mid East The ISIS has done so many terrible things in Syria and caused chaos in the Mid East The ISIS has done so many terrible things in Syria and caused chaos in the Mid East NR SC NER Human Attention (number of fixations) Task 0 5 fix (a) (b) (c) Figure 1: Human attention in different settings. NR (Normal Reading) means human reading without tasks. (Schwartz et al., 2019; Toneva and Wehbe, 2019). Prior work mainly incorporates cognitive signals by explicitly fine-tuning the pre-trained model to predict language-induced brain recordings. However, this line of work cannot give a fine- grained analysis and interpretation of why cogni- tive data is helpful for NLP. This is of great im- portance to guide future cognition-inspired NLP studies on what kind of cognitive features should be induced from cognitive data and how these cog- nitive features contribute to NLP tasks. For exam- ple, Figure 1 shows eye-tracking data from native speakers of English, where Figure 1 (a) illustrates the number of fixations during the normal …

Content truncated to 500 words