PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1939–1944 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1939 PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision Chuhan Wu† Fangzhao Wu‡ Tao Qi† Jianxun Lian‡ Yongfeng Huang† Xing Xie‡ †Department of Electronic Engineering & BNRist, Tsinghua University, Beijing 100084, China ‡Microsoft Research Asia, Beijing 100080, China {wuchuhan15,wufangzhao,taoqi.qt}@gmail.com yfhuang@tsinghua.edu.cn {Jianxun.Lian, xingx}@microsoft.com Abstract User modeling is critical for many personal- ized web services. Many existing methods model users based on their behaviors and the labeled data of target tasks. However, these methods cannot exploit useful information in unlabeled user behavior data, and their perfor- mance may be not optimal when labeled data is scarce. Motivated by pre-trained language models which are pre-trained on large-scale unlabeled corpus to empower many down- stream tasks, in this paper we propose to pre- train user models from large-scale unlabeled user behaviors data. We propose two self- supervision tasks for user model pre-training. The first one is masked behavior prediction, which can model the relatedness between his- torical behaviors. The second one is next K behavior prediction, which can model the relat- edness between past and future behaviors. The pre-trained user models are finetuned in down- stream tasks to learn task-specific user repre- sentations. Experimental results on two real- world datasets validate the effectiveness of our proposed user model pre-training method. 1 Introduction User modeling is a critical technique for many personalized web services such as personalized news and video recommendation (Okura et al., 2017; Covington et al., 2016). Many existing methods model users from their behaviors (Zhou et al., 2018; Ouyang et al., 2019). For example, Covington et al. (2016) proposed a YouTubeNet model for video recommendation, which models users from their watched videos and search tokens. Zhou et al. (2018) proposed a deep interest net- work (DIN) for click-through rate (CTR) predic- tion, which models users from user behaviors on the e-commerce platform based on their relevance to the candidate ads. Okura et al. (2017) proposed to use a GRU network for news recommendation, which models users from their clicked news. How- ever, these methods mainly rely on sufficient la- beled data to train user models, and their perfor- mance may be not optimal when training data is scarce. In addition, they only model task-specific user information and do not exploit the universal user information encoded in user behaviors. In recent years, pre-trained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) and XLNET (Yang et al., 2019) have achieved great success in many NLP tasks, such as reading comprehension and machine translation. Many language models are pre-trained on a large unlabeled corpus via self-supervision tasks such as masked LM and next sentence prediction to model the contexts (Devlin et al., 2019). These language models can learn universal language representa- tions from large unlabeled corpus and empower many different downstream tasks when the labeled data for these tasks is insufficient (Qiu et al., 2020). Motivated …

Content truncated to 500 words