Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6038–6044, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6038 Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization Siyao Li1∗, Deren Lei1∗, Pengda Qin2, William Yang Wang1 1University of California, Santa Barbara 2Beijing University of Posts and Telecommunications {siyaoli, derenlei}@ucsb.edu, qinpengda@bupt.edu.cn, william@cs.ucsb.edu Abstract Deep reinforcement learning (RL) has been a commonly-used strategy for the abstractive summarization task to address both the expo- sure bias and non-differentiable task issues. However, the conventional reward ROUGE-L simply looks for exact n-grams matches be- tween candidates and annotated references, which inevitably makes the generated sen- tences repetitive and incoherent. In this paper, instead of ROUGE-L, we explore the practica- bility of utilizing the distributional semantics to measure the matching degrees. With dis- tributional semantics, sentence-level evalua- tion can be obtained, and semantically-correct phrases can also be generated without be- ing limited to the surface form of the refer- ence sentences. Human judgments on Giga- word and CNN/Daily Mail datasets show that our proposed distributional semantics reward (DSR) has distinct superiority in capturing the lexical and compositional diversity of natural language. 1 Introduction Abstractive summarization is a task of paraphras- ing a long article with fewer words. Unlike extrac- tive summarization, abstractive summaries can in- clude tokens out of the article’s vocabulary. There exists several encoder-decoder approaches such as attention-based architecture (Bahdanau et al., 2015; Rush et al., 2015; Nallapati et al., 2016; Chopra et al., 2016), incorporating graph tech- niques (Moawad and Aref, 2012; Ganesan et al., 2010), and adding pointer generator (Vinyals et al., 2015; Nallapati et al., 2016; See et al., 2017; Bello et al., 2017). However, long sentence gener- ation suffers from exposure bias (Bahdanau et al., 2017) as the error accumulates during the decod- ing process. ∗Equal contributions. Many innovative deep RL methods (Ranzato et al., 2016; Wu et al., 2016; Paulus et al., 2018; Lamb et al., 2016) are developed to alleviate this issue by providing sentence-level feedback after generating a complete sentence, in addition to op- timal transport usage (Napoles et al., 2012). How- ever, commonly used automatic evaluation metrics for generating sentence-level rewards count exact n-grams matches and are not robust to different words that share similar meanings since the se- mantic level reward is deficient. Currently, many studies on contextualized word representations (Peters et al., 2018; Devlin et al., 2019) prove that they have a powerful capacity of reflecting distributional semantic. In this pa- per, we propose to use the distributional seman- tic reward to boost the RL-based abstractive sum- marization system. Moreover, we design several novel objective functions. Experiment results show that they outperform the conventional objectives while increasing the sentence fluency. Our main contributions are three-fold: • We are the first to introduce DSR to abstractive summarization and achieve better results than con- ventional rewards. • Unlike ROUGE, our DSR does not rely on cross- entropy …

Content truncated to 500 words