GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839 - 851 July 10-15, 2022 ©2022 Association for Computational Linguistics GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering Yoonseok Yang∗, Kyu Seok Kim∗, Minsam Kim∗, Juneyoung Park† Riiid AI Research {yoonseok.yang, kyuseok.kim, minsam.kim, juneyoung.park} @riiid.co Abstract Content-based collaborative filtering (CCF) predicts user-item interactions based on both users’ interaction history and items’ content information. Recently, pre-trained language models (PLM) have been used to extract high- quality item encodings for CCF. However, it is resource-intensive to train a PLM-based CCF model in an end-to-end (E2E) manner, since op- timization involves back-propagating through every content encoding within a given user in- teraction sequence. To tackle this issue, we propose GRAM (GRadient Accumulation for Multi-modality in CCF), which exploits the fact that a given item often appears multi- ple times within a batch of interaction his- tories. Specifically, Single-step GRAM ag- gregates each item encoding’s gradients for back-propagation, with theoretic equivalence to the standard E2E training. As an exten- sion of Single-step GRAM, we propose Multi- step GRAM, which increases the gradient update latency, achieving a further speedup with drastically less GPU memory. GRAM significantly improves training efficiency (up to 146×) on five datasets from two task domains of Knowledge Tracing and News Recommendation. Our code is available at https://github.com/yoonseok312/GRAM. 1 Introduction Collaborative filtering (CF) is a popular technique used for mining relationships between items and users. Recently, CF has been successfully applied to various tasks including Knowledge Tracing and Recommender Systems (Smith and Linden, 2017; Bennett et al., 2007; Melville et al., 2002). How- ever, conventional CF only considers item-user in- teractions, and disregards any item-/user-specific information. This leads to the so-called cold-start problem (Lam et al., 2008), where the CF model *These authors contributed equally. †This author is the corresponding author. fails to make predictions for unseen users/items, even when they resemble observed users/items. To remedy this issue, Content-based Collabo- rative Filtering (CCF) incorporates items’ content information into the item encoding. This not only addresses the cold-start problem, but also leads to significant performance improvement (Wu et al., 2019, 2021a; Lu et al., 2018). Specifically, large pre-trained language models (PLM) (Devlin et al., 2019; Brown et al., 2020) have shown great po- tential for extracting items’ content information. However, fine-tuning PLMs for CCF requires pro- hibitive computational resources in terms of (1) training time and (2) GPU memory footprint. This issue arises due to CCF’s multi-modal na- ture, where item representations are learned from both tabular user records and their textual infor- mation. As a given item appears multiple times within a batch of users’ records, its textual encod- ing needs to be computed every time it appears within the batch. Moreover, the need to store in- termediate activations for hundreds of millions of parameters in PLM requires high GPU memory footprint (Wang et al., 2020). To that end, we propose Single-step GRAM, GRadient Accumulation for Multi-modality in …

Content truncated to 500 words