Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5478 - 5489 December 7-11, 2022 ©2022 Association for Computational Linguistics Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation Yang Yu1, Fangzhao Wu2, Chuhan Wu3, Jingwei Yi1 and Qi Liu1 1University of Science and Technology of China 2Microsoft Research Asia 3Tsinghua University {yflyl613, yjw1029}@mail.ustc.edu.cn {wufangzhao, wuchuhan15}@gmail.com, qiliuql@ustc.edu.cn Abstract News recommendation is a widely adopted technique to provide personalized news feeds for the user. Recently, pre-trained language models (PLMs) have demonstrated the great capability of natural language understanding and benefited news recommendation via im- proving news modeling. However, most exist- ing works simply finetune the PLM with the news recommendation task, which may suffer from the known domain shift problem between the pre-training corpus and downstream news texts. Moreover, PLMs usually contain a large volume of parameters and have high compu- tational overhead, which imposes a great bur- den on low-latency online services. In this pa- per, we propose Tiny-NewsRec, which can im- prove both the effectiveness and the efficiency of PLM-based news recommendation. We first design a self-supervised domain-specific post- training method to better adapt the general PLM to the news domain with a contrastive matching task between news titles and news bodies. We further propose a two-stage knowl- edge distillation method to improve the effi- ciency of the large PLM-based news recom- mendation model while maintaining its per- formance. Multiple teacher models originated from different time steps of our post-training procedure are used to transfer comprehensive knowledge to the student model in both its post- training stage and finetuning stage. Extensive experiments on two real-world datasets validate the effectiveness and efficiency of our method. 1 Introduction With the explosion of information, massive news is published on online news platforms such as Mi- crosoft News and Google News (Das et al., 2007; Lavie et al., 2010), which can easily get the users overwhelmed when they try to find the information they are interested in (Okura et al., 2017; Li et al., 2020b). Many personalized news recommendation methods have been proposed to alleviate the infor- mation overload problem for users (Wang et al., 2018; Wu et al., 2019b; Zhu et al., 2019; Hu et al., 2020). Since news articles usually contain abun- dant textual content, learning high-quality news representations from news texts is one of the most critical tasks for news recommendation (Wu et al., 2020). As pre-trained language models (PLMs) have been proved to be powerful in text modeling and have empowered various NLP tasks (Devlin et al., 2019; Liu et al., 2019), a few recent works delve into employing PLMs for better news mod- eling in news recommendation (Wu et al., 2021b; Jia et al., 2021; Zhang et al., 2021b). For example, Wu et al. (2021b) propose to replace shallow NLP models such as CNN and attention network with the PLM to capture the deep contexts in news texts. However, these methods simply finetune the PLM with the news recommendation task, which may be insufficient to cope …

Content truncated to 500 words