Free Lunch for Efficient Textual Commonsense Integration in Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3759–3770 July 9-14, 2023 ©2023 Association for Computational Linguistics Free Lunch for Efficient Textual Commonsense Integration in Language Models Wanyun Cui Shanghai University of Finance and Economics cui.wanyun@shufe.edu.cn Xingran Chen University of Michigan chenxran@umich.edu Abstract Recent years have witnessed the emergence of textual commonsense knowledge bases, aimed at providing more nuanced and context- rich knowledge. The integration of external commonsense into language models has been shown to be a key enabler in advancing the state-of-the-art for a wide range of NLP tasks. However, incorporating textual commonsense descriptions is computationally expensive, as compared to encoding conventional symbolic knowledge. In this paper, we propose a method to improve its efficiency without modifying the model. We group training samples with similar commonsense descriptions into a sin- gle batch, thus reusing the encoded description across multiple samples. One key observation is that the upper bound of batch partitioning can be reduced to the classic graph k-cut problem. Consequently, we propose a spectral clustering- based algorithm to solve this problem. Exten- sive experiments illustrate that the proposed batch partitioning approach effectively reduces the computational cost while preserving perfor- mance. The efficiency improvement is more pronounced on larger datasets and on devices with more memory capacity, attesting to its practical utility for large-scale applications. 1 Introduction While pre-trained language models have made sub- stantial progress in natural language processing, they still lack certain knowledge. Thus it is criti- cal to incorporate external knowledge sources (Pe- ters et al., 2019; Zhang et al., 2019; Logan et al., 2019). Previous research has primarily focused on incorporating symbolic knowledge from structured knowledge graphs. Recently, realizing the lack of expressiveness and contextualization of sym- bolic knowledge, many forms of commonsense knowledge bases are constructed, such as if-then knowledge (Sap et al., 2019) and discourse knowl- edge (Fang et al., 2021). The integration of such textual commonsense knowledge into language models has been shown to improve the state of the art for various tasks, such as named entity recogni- tion (Wu et al., 2020) and commonsense knowledge base completion (Malaviya et al., 2020). However, integrating such commonsense knowl- edge are computationally expensive. Common- sense knowledge in text form requires more com- plex encoders (e.g. Transformer (Vaswani et al., 2017)), as opposed to the simple lookup opera- tion for discrete symbolic knowledge. The feed- forward and back-propagation process for the text encoder is significantly more computationally ex- pensive than the standalone symbolic knowledge embeddings. Therefore, it is essential to reduce the computational cost for efficient integration of textual commonsense knowledge, particularly for large-scale applications. In this paper, we propose a method to accelerate the process of incorporating textual commonsense knowledge into language models. Our approach is based on the observation that if multiple training samples in a mini-batch share the same common- sense description, the encoding for that description can be reused across those samples. In other words, we only need to encode each distinct …

Content truncated to 500 words