Question Answering as an Automatic Evaluation Metric for News Article Summarization

Proceedings of NAACL-HLT 2019, pages 3938–3948 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3938 Question Answering as an Automatic Evaluation Metric for News Article Summarization Matan Eyal1, 2, Tal Baumel1, 3, Michael Elhadad1 1Dept. Computer Science, Ben Gurion University 2IBM Research, Israel, 3Microsoft {mataney, elhadad}@cs.bgu.ac.il, tabaumel@microsoft.com Abstract Recent work in the field of automatic sum- marization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrin- sic, evaluation metric for this task, Answering Performance for Evaluation of Summaries. APES utilizes recent progress in the field of reading-comprehension to quantify the ability of a summary to answer a set of manually cre- ated questions regarding central entities in the source article. We first analyze the strength of this metric by comparing it to known man- ual evaluation metrics. We then present an end-to-end neural abstractive model that maxi- mizes APES, while increasing ROUGE scores to competitive results. 1 Introduction The task of automatic text summarization aims to produce a concise version of a source document while preserving its central information. Current summarization models are divided into two ap- proaches, extractive and abstractive. In extractive summarization, summaries are created by select- ing a collection of key sentences from the source document (e.g., Nallapati et al. (2017); Narayan et al. (2018)). Abstractive summarization, on the other hand, aims to rephrase and compress the in- put text in order to create the summary. Progress in sequence-to-sequence models (Sutskever et al., 2014) has led to recent success in abstractive sum- marization models. Current models (Nallapati et al., 2016; See et al., 2017; Paulus et al., 2017; Celikyilmaz et al., 2018) made various adjust- ments to sequence-to-sequence models to gain im- provements in ROUGE (Lin, 2004) scores. ROUGE has achieved its status as the most common method for summaries evaluation by showing high correlation to manual evaluation methods, e.g., the Pyramid method (Nenkova See et al. (2017)’s Summary: bolton will offer new contracts to emile heskey, 37, eidur gudjohnsen, 36, and adam bogdan, 27. heskey and gudjohnsen joined on short-term deals in december. eidur gudjohnsen has scored five times in the championship . APES score: 0.33 Baseline Model Summary (Encoder / Decoder / Attention / Copy / Coverage): bolton will offer new contracts to emile hes- key, 37, eidur gudjohnsen, 36, and goalkeeper adam bogdan, 27. heskey and gudjohnsen joined on short-term deals in december, and have helped neil lennon ’s side steer clear of relegation. ei- dur gudjohnsen has scored five times in the championship, as well as once in the cup this season . APES score: 0.33 Our Model (APES optimization): bolton will offer new con- tracts to emile heskey, 37, eidur gudjohnsen, 36, and goalkeeper adam bogdan, 27. heskey joined on short-term deals in decem- ber, and have helped neil lennon ’s side steer clear of relegation. eidur gudjohnsen has scored five times in the championship, as well as once in the cup this season. lennon has also fined mid- …

Content truncated to 500 words