Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1796–1802 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1796 Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings Abdul Moeed†, Yang An†, Gerhard Hagerer†, Georg Groh Research Group Social Computing, Department of Informatics Technical University of Munich, Germany {abd.moeed, yang.an, gerhard.hagerer}@tum.de, grohg@in.tum.de † These are equally first authors and appear in random order. Abstract With the explosive growth in textual data, it is becoming increasingly important to summarize text automatically. Recently, generative language models have shown promise in abstractive text summarization tasks. Since these models rephrase text and thus use similar but different words as found in the summarized text, existing metrics such as ROUGE that use n-gram overlap may not be optimal. Therefore we evaluate two embedding-based evaluation metrics that are applicable to abstractive summarization: Fr´echet embedding distance, which has been introduced recently, and angular embedding similarity, which is our proposed metric. To demonstrate the utility of both metrics, we analyze the headline generation capacity of two state-of-the-art language models: GPT-2 and ULMFiT. In particular AES shows close relation with human judgments in our experiments and has overall better correlations with them compared to ROUGE. To provide reproducibility, the source code plus human assessments of our experiments is available on GitHub1. Keywords: Evaluation Methodologies, Language Modelling, Natural Language Generation, Summarization, Textual Entailment and Paraphrasing, Statistical and Machine Learning Methods 1. Introduction The recent development of generative language models (LMs) is leading to new capabilities regarding the quality of text generation (Radford et al., 2019). This also holds true for tasks such as abstractive summarization, which is re- lated to the language model generating summaries de nou- veau and paraphrasing the text in its own words (Moratanch and Chitrakala, 2016). This is of high importance consid- ering the large and always increasing amount of available texts and their relevance for humans. An advantage of abstractive summarization is its superior readability (Hsu et al., 2018) compared to extractive sum- marization where keywords from the text are extracted and rearranged (Lin and Hovy, 2003). This benefit can be used for generating realistic headlines (Takase et al., 2016; See et al., 2017). However, it remains a challenge to find a faithful evaluation metric. ROUGE (Lin, 2004) - a standard perfor- mance metric for extractive summarization - is not always ideal for abstractive summarization, since readability is not taken into account (Paulus et al., 2017). Instead, it only ac- counts for n-gram overlap which is a problem for use cases when summaries rephrase the respective content using dif- ferent but similar words. To address this problem, pre-trained semantic similarity embeddings such as InferSent (Conneau et al., 2017) have been used successfully to evaluate the quality of GAN- based text generation (Semeniuta et al., 2018). Therefore, the concept of the Fr´echet distance (Heusel et al., 2017), which is a well-known procedure for computer vision, is successfully applied for text generation as well. Due to the novelty …

Content truncated to 500 words