Better Rewards Yield Better Summaries: Learning to Summarise Without References

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3110–3120, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3110 Better Rewards Yield Better Summaries: Learning to Summarise Without References Florian B¨ohm1, Yang Gao1∗, Christian M. Meyer1, Ori Shapira2, Ido Dagan2, and Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab, Technische Universit¨at Darmstadt, Germany 2Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel https://www.ukp.tu-darmstadt.de/ yang.gao@rhul.ac.uk, obspp18@gmail.com, dagan@cs.biu.ac.il Abstract Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, be- cause they directly use ROUGE as the re- wards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better re- ward function that can guide RL to generate human-appealing summaries, we learn a re- ward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as in- put. Hence, once trained, it can be used to train RL-based summarisation systems with- out using any reference summaries. We show that our learned rewards have signif- icantly higher correlation with human rat- ings than previous approaches. Human eval- uation experiments show that, compared to the state-of-the-art supervised-learning sys- tems and ROUGE-as-rewards RL summarisa- tion systems, the RL systems using our learned rewards during training generate summaries with higher human ratings. The learned re- ward function and our source code are avail- able at https://github.com/yg211/ summary-reward-no-reference. 1 Introduction Document summarisation aims at generating a summary for a long document or multiple docu- ments on the same topic. Reinforcement Learning (RL) becomes an increasingly popular technique to build document summarisation systems in re- cent years (Chen and Bansal, 2018; Narayan et al., 2018b; Dong et al., 2018). Compared to the su- pervised learning paradigm, which “pushes” the summariser to reproduce the reference summaries * Since June 2019, Yang Gao is affiliated with Dept. of Computer Science, Royal Holloway, University of London. at training time, RL directly optimises the sum- mariser to maximise the rewards, which measure the quality of the generated summaries. The ROUGE metrics (Lin, 2004b) are the most widely used rewards in training RL-based sum- marisation systems. ROUGE measures the quality of a generated summary by counting how many n-grams in the reference summaries appear in the generated summary. ROUGE correlates well with human judgements at system level (Lin, 2004a; Louis and Nenkova, 2013), i.e. by aggregating sys- tem summaries’ ROUGE scores across multiple input documents, we can reliably rank summari- sation systems by their quality. However, ROUGE performs poorly at summary level: given multiple summaries for the same input document, ROUGE can hardly distinguish the “good” summaries from the “mediocre” and “bad” ones (Novikova et al., 2017). Because existing RL-based summarisation systems rely on summary-level ROUGE scores to guide the optimisation direction, the poor perfor- mance of ROUGE at summary level severely mis- leads the RL agents. The reliability of ROUGE as RL reward is further challenged by the fact that …

Content truncated to 500 words