MTRec: Multi-Task Learning over BERT for News Recommendation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2663 - 2669 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MTRec: Multi-Task Learning over BERT for News Recommendation Qiwei Bi1*, Jian Li2*, Lifeng Shang2, Xin Jiang2, Qun Liu2, Hanfang Yang3,1† 1School of Statistics, Renmin University of China 2Huawei Noah’s Ark Lab 3Center for Applied Statistics, Renmin University of China {bqw, hyang}@ruc.edu.cn {lijian703, shang.lifeng, jiang.xin, qun.liu}@huawei.com Abstract Existing news recommendation methods usu- ally learn news representations solely based on news titles. To sufficiently utilize other fields of news information such as category and en- tities, some methods treat each field as an ad- ditional feature and combine different feature vectors with attentive pooling. With the adop- tion of large pre-trained models like BERT in news recommendation, the above way to incor- porate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding. In this paper, we propose a multi-task learning framework to incorporate the multi-field infor- mation into BERT, which improves its news encoding capability. Besides, we modify the gradients of different tasks based on their gra- dient conflicts, which further boosts the model performance. Extensive experiments on the MIND news recommendation benchmark show the effectiveness of our approach. 1 Introduction Online News platforms such as Google News and MSN News have become a prevalent way for users to access news information (Das et al., 2007). To al- leviate information overload and improve the read- ing experience, personalized news recommenda- tion has become an essential part of these plat- forms (Liu et al., 2010; Phelan et al., 2011). Traditional Recommendation models focus on modeling feature interactions (Rendle, 2012; Cheng et al., 2016; Guo et al., 2017; Wang et al., 2017). Accurate modeling of news and users is critical for news representation. Previous neural methods usually learn news representation vectors solely based on news titles and then learn a user rep- resentation by aggregating the previously browsed * Equal contribution. This work was done when Qiwei Bi was an intern at Huawei Noah’s Ark Lab. † Corresponding author Title: Kate Middleton’s Best Hairstyles Through the Years (Category: Lifestyle) Kate Middleton Years Transformer Attention Lifestyle Topic Emb. Dense Kate Middleton Entity Emb. Attention Attention 𝒓𝒕 𝒓𝑐 𝒓𝑒 ෤𝒓 News Embedding Title Category Entities Figure 1: Traditional way to incorporate multi-field news information with attentive multi-field learning. news via sequential or attentive models (Okura et al., 2017; An et al., 2019; Wu et al., 2019d). Though effective, these methods only utilize the title information and neglect other valuable news in- formation such as categories and entities, which we call multi-field information. To fully utilize this in- formation, as shown in Figure 1, existing methods usually transform each field of information (e.g., title, category, and entities) into a feature vector and combine different representations via attentive multi-field learning (Wu et al., 2019a, 2021a). With the widespread use of large pre-trained language models, news recommendations start to adopt BERT (Devlin et …

Content truncated to 500 words