ROUGE

This paper describes an empirical study of the “In- formation Synthesis” task, defined as the process of (given a complex information need) extracting, or- ganizing and inter-relating the pieces of …...

We consider the evaluation problem in Natural Language Generation (NLG) and present results for evaluating several NLG systems with similar functionality, includ- ing a knowledge-based generator and sev- eral statistical …...

ParaEval is an automated evaluation method for comparing reference and peer summaries. It facilitates a tiered- comparison strategy where recall-oriented global optimal and local greedy searches for paraphrase matching are …...

This paper presents a strategy to generate ge- neric summary of documents using Probabilistic Latent Semantic Indexing. Generally a docu- ment contains several topics rather than a single one. Summaries …...

In this paper, we analyze the state of cur- rent human and automatic evaluation of topic-focused summarization in the Docu- ment Understanding Conference main task for 2005-2007. The analyses show …...

This paper addresses the problem of ex- tracting the most important facts from a news article. Our approach uses syntac- tic, semantic, and general statistical fea- tures to identify the …...

We present an exploration of generative prob- abilistic models for multi-document summa- rization. Beginning with a simple word fre- quency based model (Nenkova and Vander- wende, 2005), we construct a …...

Scoring sentences in documents given ab- stract summaries created by humans is im- portant in extractive multi-document sum- marization. In this paper, we formulate ex- tractive summarization as a two …...

AbstractThis paper presents a new algorithm for automatic summarization of specialized texts combining terminological and semantic resources: a term extractor and an ontology. The term extractor provides the list of …...

This paper analyzes the topic identification stage of single-document automatic text sum- marization across four different domains, con- sisting of newswire, literary, scientific and le- gal documents. We present a …...

Extractive methods for multi-document sum- marization are mainly governed by informa- tion overlap, coherence, and content con- straints. We present an unsupervised proba- bilistic approach to model the hidden abstract …...

We learn a joint model of sentence extraction and compression for multi-document summa- rization. Our model scores candidate sum- maries according to a combined linear model whose features factor over …...

This paper proposes a method for automat- ically generating summaries taking into account the information in which users may be interested. Our approach relies on existing model summaries from tourist …...

AbstractVector space models benefit from using an outside corpus to train the model. It is, however, unclear what constitutes a good training corpus. We have investigated the effect on summary …...

We present a new approach to the problem of automatic text summarization called Au- tomatic Summarization using Reinforcement Learning (ASRL) in this paper, which models the process of constructing a …...

Multi-document summarization involves many aspects of content selection and sur- face realization. The summaries must be informative, succinct, grammatical, and obey stylistic writing conventions. We present a method where such …...

Extractive summarization typically uses sen- tences as summarization units. In contrast, joint compression and summarization can use smaller units such as words and phrases, re- sulting in summaries containing more …...

ion through event pattern clustering Enrique Alfonseca Google Inc. ealfonseca@google.com Daniele Pighin Google Inc. biondo@google.com Guillermo Garrido∗ NLP & IR Group at UNED ggarrido@lsi.uned.es Abstract This paper presents HEADY: a …...

ROUGE is a widely adopted, automatic evaluation measure for text summariza- tion. While it has been shown to corre- late well with human judgements, it is bi- ased towards surface …...

Integer Linear Programming (ILP) based sum- marization methods have been widely adopted recently because of their state-of-the-art per- formance. This paper proposes two new mod- ifications in this framework for …...

Opinion summarization is the task of pro- ducing the summary of a text, such that the summary also preserves the sentiment of the text. Opinion Summarization is thus a trade-off …...

We provide an analysis of current evalua- tion methodologies applied to summariza- tion metrics and identify the following ar- eas of concern: (1) movement away from evaluation by correlation with …...

AbstractIn this paper we report a comparison of various techniques for single-document extractive summarization under strict length budgets, which is a common commercial use case (e.g. summarization of news articles …...

AbstractNatural language generation (NLG) is the task of generating natural language from a meaning representation. Current rule-based approaches require domain-specific and manually constructed linguistic resources, while most machine-learning based approaches …...

We present a discriminative model for single-document summarization that integrally combines compression and anaphoricity constraints. Our model selects textual units to include in the summary based on a rich set …...

The timeline generation task summarises an entity’s biography by selecting stories representing key events from a large pool of relevant documents. This paper ad- dresses the lack of a standard …...

AbstractMulti-document summarization provides users with a short text that summarizes the information in a set of related documents. This paper introduces affinity-preserving random walk to the summarization task, which preserves …...

AbstractWe propose a novel metric for evaluating summary content coverage. The evaluation framework follows the Pyramid approach to measure how many summarization content units, considered important by human annotators, are …...

AbstractA major proportion of a text summary includes important entities found in the original text. These entities build up the topic of the summary. Moreover, they hold commonsense information once …...

AbstractAutomatic evaluation systems in the field of automatic summarization have been relying on the availability of gold standard summaries for over ten years. Gold standard summaries are expensive to obtain …...

AbstractMotivated by recent findings on the probabilistic modeling of acceptability judgments, we propose syntactic log-odds ratio (SLOR), a normalized language model score, as a metric for referenceless fluency evaluation of …...

AbstractIn sentence compression, the task of shortening sentences while retaining the original meaning, models tend to be trained on large corpora containing pairs of verbose and compressed sentences. To remove …...

AbstractWe propose a global optimization method under length constraint (GOLC) for neural text summarization models. GOLC increases the probabilities of generating summaries that have high evaluation scores, ROUGE in this …...

AbstractAutomatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that …...

AbstractNeural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder …...

AbstractVarious Seq2Seq learning models designed for machine translation were applied for abstractive summarization task recently. Despite these models provide high ROUGE scores, they are limited to generate comprehensive summaries with …...

AbstractTranscripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models for generating summaries unfocused. We develop an abstractive meeting summarizer from …...

AbstractRecent work in the field of automatic summarization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrinsic, evaluation metric for this task, …...

AbstractOne of the most common metrics to automatically evaluate opinion summaries is ROUGE, a metric developed for text summarisation. ROUGE counts the overlap of word or word units between a …...

AbstractNeural architectures based on self-attention, such as Transformers, recently attracted interest from the research community, and obtained significant improvements over the state of the art in several tasks. We explore …...

AbstractFor evaluating machine-generated texts, automatic methods hold the promise of avoiding collection of human judgments, which can be expensive and time-consuming. The most common automatic metrics, like BLEU and ROUGE, …...

AbstractAutomatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text qualityis represented in a variety of dimensions ranging from grammaticality to …...

AbstractWe present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the …...

AbstractText generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with …...

AbstractEntities are the major proportion and build up the topic of text summaries. Although existing text summarization models can produce promising results of automatic metrics, for example, ROUGE, it is …...

AbstractAutomatic sentence summarization produces a shorter version of a sentence, while preserving its most important information. A good summary is characterized by language fluency and high information overlap with the …...

AbstractCommonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage …...

AbstractOne of the fundamental goals of artificial intelligence is to build computer-based expert systems. Inferring clinical diagnoses to generate a clinical assessment during a patient encounter is a crucial step …...

AbstractEfficient document summarization requires evaluation measures that can not only rank a set of systems based on an average score, but also highlight which individual summary is better than another. …...

AbstractSequence-to-sequence models for abstractive summarization have been studied extensively, yet the generated summaries commonly suffer from fabricated content, and are often found to be near-extractive. We argue that, to address …...

AbstractSo far work on automatic summarization has dealt primarily with English data. Accordingly, evaluation methods were primarily developed with this language in mind. In our work, we present experiments of …...

AbstractQuestion generation (QG) attempts to solve the inverse of question answering (QA) problem by generating a natural language question given a document and an answer. While sequence to sequence neural …...

AbstractDeep learning has led to significant improvement in text summarization with various methods investigated and improved ROUGE scores reported over the years. However, gaps still exist between summaries produced by …...

In extractive summarization, searching for the most salient sentences to generate summaries can be resolved by applying meta-heuristic optimization algorithms including genetic algorithms (GAs), par- ticle swarm optimization (PSO), etc. …...

AbstractThe quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how …...

AbstractSentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents. However, constructing a coherent and informative summary is difficult using a pre-trained BERT-based …...

AbstractPodcast episodes often contain material extraneous to the main content, such as advertisements, interleaved within the audio and the written descriptions. We present classifiers that leverage both textual and listening …...

AbstractThe quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise …...

AbstractRecent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch …...

AbstractSummarisation of reviews aims at compressing opinions expressed in multiple review documents into a concise form while still covering the key opinions. Despite the advancement in summarisation models, evaluation metrics …...

AbstractSentence fusion is a conditional generation task that merges several related sentences into a coherent one, which can be deemed as a summary sentence. The importance of sentence fusion has …...

AbstractProfessional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on …...

AbstractThe availability of large-scale datasets has driven the development of neural models that create generic summaries from single or multiple documents. In this work we consider query focused summarization (QFS), …...

AbstractWe present GeSERA, an open-source improved version of SERA for evaluating automatic extractive and abstractive summaries from the general domain. SERA is based on a search engine that compares candidate …...

State-of-the-art summarization systems are trained and evaluated on massive datasets scraped from the web. Despite their preva- lence, we know very little about the underly- ing characteristics (data noise, summarization …...

Text generation from semantic parses is to generate textual descriptions for formal rep- resentation inputs such as logic forms and SQL queries. This is challenging due to two reasons: (1) …...

AbstractSummarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work …...

AbstractA desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with …...

AbstractReference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary’s information quality …...

AbstractAutomatic text generation based on neural language models has achieved performance levels that make the generated text almost indistinguishable from those written by humans. Despite the value that text generation …...

AbstractSequential abstractive neural summarizers often do not use the underlying structure in the input article or dependencies between the input sentences. This structure is essential to integrate and consolidate information …...

In this paper, we present a pipeline for auto- matic minuting. This pipeline is an end-to-end system for minuting the multiparty dialogues of meetings. It provides multilingual com- munication and …...

AbstractA law practitioner has to go through numerous lengthy legal case proceedings for their practices of various categories, such as land dispute, corruption, etc. Hence, it is important to summarize …...

AbstractNatural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to …...

AbstractRecent research on code summarization relies on the structural information from the abstract syntax tree (AST) of source codes. It is, however, questionable whether it is the most effective to …...

AbstractDialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be …...

AbstractTransformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE, a novel dynamic latent extraction approach for …...

AbstractWe tackle the problem of neural headline generation in a low-resource setting, where only limited amount of data is available to train a model. We compare the ideal high-resource scenario …...

AbstractAutomatic dialogue summarization is a task used to succinctly summarize a dialogue transcript while correctly linking the speakers and their speech, which distinguishes this task from a conventional document summarization. …...

AbstractThe automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as …...

AbstractDocument structure is critical for efficient information consumption. However, it is challenging to encode it efficiently into the modern Transformer architecture. In this work, we present HIBRIDS, which injects Hierarchical …...

AbstractMental health remains a significant challenge of public health worldwide. With increasing popularity of online platforms, many use the platforms to share their mental health conditions, express their feelings, and …...

AbstractCode-mixing is the phenomena of mixing various linguistic units such as paragraphs, sentences, phrases, words, etc., of one language with that of the other language in any text. This code-mixing …...

AbstractWe present a multi-task learning framework for cross-lingual abstractive summarization to augment training data. Recent studies constructed pseudo cross-lingual abstractive summarization data to train their neural encoder-decoders. Meanwhile, we introduce …...

AbstractText clustering methods were traditionally incorporated into multi-document summarization (MDS) as a means for coping with considerable information repetition. Particularly, clusters were leveraged to indicate information saliency as well as …...

AbstractHow reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation …...

AbstractIt has been the norm for a long time to evaluate automated summarization tasks using the popular ROUGE metric. Although several studies in the past have highlighted the limitations of …...

AbstractCanonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. …...

AbstractSequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search …...

AbstractMany scientific papers such as those in arXiv and PubMed data collections have abstracts with varying lengths of 50-1000 words and average length of approximately 200 words, where longer abstracts …...

AbstractUnsupervised extractive summarization has recently gained importance since it does not require labeled data. Among unsupervised methods, graph-based approaches have achieved outstanding results. These methods represent each document by a …...

AbstractRecent abstractive conversation summarization systems generally rely on large-scale datasets with annotated summaries. However, collecting and annotating these conversations can be a time-consuming and labor-intensive task. To address this issue, …...

AbstractWe present CrossSum, a large-scale cross-lingual summarization dataset comprising 1.68 million article-summary samples in 1,500+ language pairs. We create CrossSum by aligning parallel articles written in different languages via cross-lingual …...

AbstractDialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations because they do not take into account the …...

AbstractArgument summarisation is a promising but currently under-explored field. Recent work has aimed to provide textual summaries in the form of concise and salient short texts, i.e., key points (KPs), …...

AbstractTwo-step approaches, in which summary candidates are generated-then-reranked to return a single summary, can improve ROUGE scores over the standard single-step approach. Yet, standard decoding methods (i.e., beam search, nucleus …...

AbstractCorrect, comparable, and reproducible model evaluation is essential for progress in machine learning. Over twenty years, thousands of language and vision models have been evaluated with a popular metric called …...

AbstractPre-trained transformers are popular in state-of-the-art dialogue generation (DG) systems. Such language models are, however, vulnerable to various adversarial samples as studied in traditional tasks such as text classification, which …...