Better Language Model with Hypernym Class Prediction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1352 - 1362 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Better Language Model with Hypernym Class Prediction He Bai∗ University of Waterloo he.bai@uwaterloo.ca Tong Wang Microsoft Research Tong.Wang@microsoft.com Alessandro Sordoni Microsoft Research alsordon@microsoft.com Peng Shi University of Waterloo peng.shi@uwaterloo.ca Abstract Class-based language models (LMs) have been long devised to address context sparsity in n-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a com- mon WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves per- plexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and ARXIV. Our analysis shows that the performance improve- ment is achieved without sacrificing perfor- mance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale. 1 Introduction Over the course of the past decades, language mod- eling (LM) has transitioned from n-gram to neu- ral models (Bengio et al., 2003; Mnih and Hinton, 2007; Devlin et al., 2019; Brown et al., 2020). Per- formance improvement of today’s neural LMs is often achieved at the cost of increased computa- tional resources. For example, to capture long-term dependencies, various extensions of Transformer- based LMs have been proposed (Dai et al., 2019; Rae et al., 2020). These modifications bring about significant improvements on held-out perplexity, but training cost also increases significantly due to large GPU memory consumption and more compu- tations at each training step. In parallel, alternative training strategies have also been proposed (Guu et al., 2020; Ziegler ∗Most of the work was done during the internship at Microsoft Research. Code: https://github.com/ richardbaihe/robustLM.git A final torch used to enter Empire Stadium that was made of stainless steel and powered by a magnesium candle Original Text: Replaced with hypernym class: A final instrumentality.n.03 used to enter Empire structure.n.01 that was made of alloy.n.01 alloy.n.01 and powered by a metallic_element.n.01 instrumentality.n.03 Figure 1: An example of word prediction training text and hypernym class prediction training text. and Rush, 2019; Deng et al., 2020). In this pa- per, we explore the effectiveness of class-based language models (CLMs, Brown et al. 1992) in the context of neural LMs. CLMs group indi- vidual words into coarser-grained classes and has proven effective in alleviating context sparsity in n-gram LMs (Dagan et al., 1999). It has been also used to improve computational efficiency in neural LMs (Morin and Bengio, 2005; Grave et al., 2017a). More recently, Levine et al. (2020) pretrain masked LMs (Devlin et al., 2019) by predicting WordNet supersense labels. However, the work focuses on word-sense disambiguation tasks and …

Content truncated to 500 words