Language Model Pre-Training with Sparse Latent Typing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1480 - 1494 December 7-11, 2022 ©2022 Association for Computational Linguistics Language Model Pre-Training with Sparse Latent Typing Liliang Ren1∗, Zixuan Zhang1∗, Han Wang2, Clare R. Voss3, Chengxiang Zhai1, Heng Ji1 1University of Illinois at Urbana-Champaign, 2Amazon Alexa, 3US Army Research Laboratory {liliang3, zixuan11, czhai, hengji}@illinois.edu wnghn@amazon.com, clare.r.voss.civ@army.mil Abstract Modern large-scale Pre-trained Language Mod- els (PLMs) have achieved tremendous success on a wide range of downstream tasks. How- ever, most of the LM pre-training objectives only focus on text reconstruction, but have not sought to learn latent-level interpretable rep- resentations of sentences. In this paper, we manage to push the language models to obtain a deeper understanding of sentences by propos- ing a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge. Besides, the lan- guage model pre-trained with such an objective also significantly improves Information Extrac- tion related downstream tasks in both super- vised and few-shot settings. Our code is pub- licly available at https://github.com/renll/ SparseLT. 1 Introduction Transformer-based Pre-trained Language Models (PLMs) have achieved significant success on a wide range of NLP tasks. However, typical pre-training objectives for PLMs only focus on teaching the model to directly reconstruct text-level words or sentences, but have not sought to obtain deeper sen- tence understanding by learning latent-level inter- pretable representations. For example, transformer- decoder models like the OpenAI GPT series (Rad- ford et al., 2018, 2019; Brown et al., 2020) adopt ∗ Equal contribution. Listing order is random. Liliang proposed and implemented the architecture designs and the training objectives of Sparse Latent Typing (SLT), and he also conducted extensive experiments for pre-training, few-shot evaluation and the analyses. Zixuan designed the language model pre-training pipeline for SLT, built the initial training codebase and conducted experiments for pre-training and su- pervised evaluation. Both of the authors initially came up with the same project goal of encouraging the model to sparsely select sentence-level key words during pre-training. Pretrained LM A Montana helicopter company transported 40 people out from the flooding. Decoder “Montana” “helicopter” “transported” “40 people” “flooding” Type: 1 Type: 2 Type: 3 Type: 2 Type: 3 <s> A Montana helicopter … “A” Type: 0 “out” “from” Type: 0 Type: 0 Selected Typed Tokens Sentence Reconstruction Reconstruction Loss Sparsity Loss + Downstream IE tasks Fine-tune Figure 1: A general illustration of our approach to teach pre-trained language model to extract sentence-level keywords with latent type representations in a com- pletely self-supervised manner. the task of language modeling for pre-training, and transformer-encoder models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are trained by predicting the masked tokens within a sentence. Both of these training objectives merely train the models to recover the masked tokens or predict the next words or sentences, while ignoring …

Content truncated to 500 words