Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1838–1848, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads Ji He∗, Mari Ostendorf∗, Xiaodong He†, Jianshu Chen†, Jianfeng Gao†, Lihong Li†, Li Deng† ∗Department of Electrical Engineering, University of Washington, Seattle, WA 98195, USA {jvking, ostendor}@uw.edu †Microsoft Research, Redmond, WA 98052, USA {xiaohe, jianshuc, jfgao, lihongli, deng}@microsoft.com Abstract We introduce an online popularity prediction and tracking task as a benchmark task for rein- forcement learning with a combinatorial, nat- ural language action space. A specified num- ber of discussion threads predicted to be pop- ular are recommended, chosen from a fixed window of recent comments to track. Novel deep reinforcement learning architectures are studied for effective modeling of the value function associated with actions comprised of interdependent sub-actions. The proposed model, which represents dependence between sub-actions through a bi-directional LSTM, gives the best performance across different ex- perimental configurations and domains, and it also generalizes well with varying numbers of recommendation requests. 1 Introduction This paper is concerned with learning policies for sequential decision-making tasks, where a system takes actions given options characterized by natu- ral language with the goal of maximizing a long- term reward. More specifically, we consider tasks with a combinatorial action space, where each ac- tion is a set of multiple interdependent sub-actions. The problem of a combinatorial natural language ac- tion space arises in many applications. For example, in real-time news feed recommendation, a user may want to read diverse topics of interest, and an ac- tion (i.e. recommendation) from the computer agent would consist of a set of news articles that are not all similar in topics (Yue and Guestrin, 2011). In adver- tisement placement, an action is a selection of sev- eral ads to display, and bundling with complemen- tary products might receive higher click-through- rate than displaying all similar popular products. In this work, we consider Reddit popularity pre- diction, which is similar to newsfeed recommenda- tion but different in two respects. First, our goal is not to make recommendations based on an indi- vidual’s preferences, but instead based on the antic- ipated long-term interest level of a broad group of readers from a target community. Second, we try to predict rather than detect popularity. Unlike individ- ual interests, community interest level is not often immediately clear; there is a time lag before the level of interest starts to take off. Here, the goal is for the recommendation system to identify and track writ- ten documents (e.g. news articles, comments in dis- cussion forum threads, or scientific articles) in real time – attempting to identify hot updates before they become hot to keep the reader at the leading edge. The premise is that the user’s bandwidth is limited, and only a limited number of things can be recom- mended out of several possibilities. In our experi- mental work, we use discussion forum text, where …

Content truncated to 500 words