BERTScore

AbstractMany automatic evaluation metrics have been proposed to score the overall quality of a response in open-domain dialogue. Generally, the overall quality is comprised of various aspects, such as relevancy, …...

AbstractEvaluating image captions is very challenging partially due to the fact that there are multiple correct captions for every single image. Most of the existing one-to-one metrics operate by penalizing …...

In recent years, the quality of machine transla- tion has significantly improved, and it is con- sidered that translation of a novel becomes possible. However, when performing transla- tion for …...

AbstractBERTScore, a recently proposed automatic metric for machine translation quality, uses BERT, a large pre-trained language model to evaluate candidate translations with respect to a gold translation. Taking advantage of …...

AbstractRecent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch …...

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first at- tempt to systematically quantify their panlin- guistic efficacy. We take a …...

AbstractEvaluation metrics are a key ingredient for progress of text generation systems. In recent years, several BERT-based evaluation metrics have been proposed (including BERTScore, MoverScore, BLEURT, etc.) which correlate much …...

In this paper, we define a sentence re- generation task: re-generate a sentence given a set of sentences that cover it. Due to the absence of a dataset to perform …...

AbstractCurrent metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate …...

AbstractThis paper presents our submission for the shared task on isometric neural machine translation in International Conference on Spoken Language Translation (IWSLT). There are numerous state-of-art models for translation problems. …...

AbstractIn recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to …...

AbstractWe explore Interactive Post-Editing (IPE) models for human-in-loop translation to help correct translation errors and rephrase it with a desired style variation. We specifically study verbosity for style variations and …...

AbstractThe evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. However, these benchmarks are mostly from similar …...

ive Summaries Tim Fischer Steffen Remus Chris Biemann Language Technology Group Department of Informatics Universität Hamburg, Germany {firstname.lastname}@uni-hamburg.de Abstract Recent abstractive summarization systems fail to generate factually consistent – faithful …...

AbstractReal human conversation data are complicated, heterogeneous, and noisy, from which building open-domain dialogue systems remains a challenging task. In fact, such dialogue data still contains a wealth of information …...

AbstractPretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence generation tasks (e.g., machine translation and text summarization) due to their better correlation with …...

AbstractMultilingual Neural Machine Translation (MNMT) models allow to translate across multiple languages based on only one system. We study the quality of a domain-adapted MNMT model in the medical domain …...

AbstractWe approach the task of assessing the suitability of a source text for translation by transferring the knowledge from established MT evaluation metrics to a model able to predict MT …...