Towards the Use of Deep Reinforcement Learning with Global Policy for Query-based Extractive Summarisation

Towards the Use of Deep Reinforcement Learning with Global Policy For Query-based Extractive Summarisation⇤ Diego Moll´a Department of Computing Macquarie University Sydney diego.molla-aliod@mq.edu.au Abstract Supervised approaches for text summari- sation suffer from the problem of mis- match between the target labels/scores of individual sentences and the evaluation score of the final summary. Reinforcement learning can solve this problem by pro- viding a learning mechanism that uses the score of the final summary as a guide to determine the decisions made at the time of selection of each sentence. In this paper we present a proof-of-concept approach that applies a policy-gradient algorithm to learn a stochastic policy using an undis- counted reward. The method has been ap- plied to a policy consisting of a simple neural network and simple features. The resulting deep reinforcement learning sys- tem is able to learn a global policy and ob- tain encouraging results. 1 Introduction Common supervised machine learning approaches to extractive summarisation attempt to label indi- vidual text extracts (usually sentences or phrases; in this paper we will use sentences). In a sub- sequent stage, a summary is generated based on the predicted labels of the individual sentences and other factors such as redundancy of information. The process of obtaining the annotated data can be complex. Data sets often contain complete summaries written manually. Well-known exam- ples of data sets of this type are the DUC and TAC data sets (Dang, 2006, 2008). In such cases the task of labelling individual sentences is not straightforward and needs to be derived from the full summaries. Alternatively, annotations can be ⇤ Code available at https://github.com/ dmollaaliod/alta2017-rl obtained through highlights made by the annota- tors (Woodsend and Lapata, 2010, for example). Regardless of the means used to annotate indi- vidual sentences, the final evaluation of the system compares the output summary with a set of target summaries, either by using human judges or auto- matically by using packages such as ROUGE (Lin, 2004). However, machine learning approaches de- signed to minimise the prediction error of individ- ual sentences would not necessarily minimise the prediction error of the final summary evaluation metric. In this paper we propose a proof-of-concept method that uses reinforcement learning with global policy as a means to use the ROUGE L evaluation of the final summary directly in the training process. Section 2 introduces reinforce- ment learning and mentions past work on the use of reinforcement learning for summarisation. Sec- tion 3 describes our proposal for the use of re- inforcement learning for query-based summarisa- tion. Section 4 presents the results of our experi- ments, and Section 5 concludes this paper. 2 Reinforcement Learning Reinforcement Learning (RL) is a machine learn- ing approach that is designed to train systems that aim to maximise a long-term goal, even when there is no knowledge (or little knowledge) of the impact of the individual decisions that are made to achieve the goal. A RL task (Figure 1) consists of an environment that can be observed and can …

Content truncated to 500 words