Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3246–3256, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3246 Answers Unite! Unsupervised Metrics for Reinforced Summarization Models Thomas Scialom⋆‡, Sylvain Lamprier‡, Benjamin Piwowarski⋄‡, Jacopo Staiano⋆ ⋄ CNRS, France ‡ Sorbonne Universit´e, CNRS, LIP6, F-75005 Paris, France ⋆ reciTAL, Paris, France {thomas,jacopo}@recital.ai {sylvain.lamprier,benjamin.piwowarski}@lip6.fr Abstract Abstractive summarization approaches based on Reinforcement Learning (RL) have re- cently been proposed to overcome classi- cal likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similar- ity as well as from suboptimal accounting for fluency and readability of the generated ab- stracts. We thus explore and propose alterna- tive evaluation measures: the reported human- evaluation analysis shows that the proposed metrics, based on Question Answering, favor- ably compares to ROUGE – with the addi- tional property of not requiring reference sum- maries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as a reward. 1 Introduction Summarization systems aim at generating relevant and informative summaries given a variable-length text as input. They can be roughly divided un- der two main categories, those adopting an extrac- tive approach, i.e. identifying the most informative pieces from the input text and concatenating them to form the output summary; and those producing abstractive summaries, i.e. generating an output text whose tokens are not necessarily present in the input text. While closer to human summarization, abstrac- tive summarization is a much harder task and the need for faithful evaluation metrics is crucial to measure and drive the progress of such systems. The standard for evaluation of summarization sys- tems is ROUGE (Lin, 2004): this metric can be considered as an adaptation of BLEU (Papineni et al., 2002), a scoring method for evaluation of machine translation systems; both based on n- gram co-occurrences, the latter favors precision while the former emphasizes recall. Recent research works (Paulus et al., 2017; Pa- sunuru and Bansal, 2018; Arumae and Liu, 2019) have proposed to use evaluation metrics – and ROUGE in particular – to learn the model pa- rameters through Reinforcement Learning (RL) techniques. This makes the choice of a good evaluation metric even more important. Unfortu- nately, ROUGE is known to incur several prob- lems: in particular, its poor accounting for flu- ency and readability of the generated abstracts, as well as its bias towards lexical similarity (Ng and Abrecht, 2015). To emphasize the latter point, since ROUGE evaluates a summary against given human references, summarization models incur the risk of being unfairly penalized: a high qual- ity summary might still have very few tokens/n- grams in common with the reference it is evaluated against. In this work, we propose to overcome n-gram matching based metrics, …

Content truncated to 500 words