BLEURT
Related Papers
BLEURT: Learning Robust Metrics for Text Generation
AbstractText generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with …...
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
AbstractThe quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem. This paper describes our contribution to …...
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
AbstractEvaluation metrics are a key ingredient for progress of text generation systems. In recent years, several BERT-based evaluation metrics have been proposed (including BERTScore, MoverScore, BLEURT, etc.) which correlate much …...
Learning Compact Metrics for MT
AbstractRecent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use …...
Logic-Consistency Text Generation from Semantic Parses
Text generation from semantic parses is to generate textual descriptions for formal rep- resentation inputs such as logic forms and SQL queries. This is challenging due to two reasons: (1) …...
RoBLEURT Submission for WMT2021 Metrics Task
AbstractIn this paper, we present our submission to Shared Metrics Task: RoBLEURT (Robustly Optimizing the training of BLEURT). After investigating the recent advances of trainable metrics, we conclude several aspects …...
Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality?
AbstractNeural machine translation (NMT) is often criticized for failures that happenwithout awareness. The lack of competency awareness makes NMT untrustworthy. This is in sharp contrast to human translators who give …...
DEMETR: Diagnosing Evaluation Metrics for Translation
AbstractWhile machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced …...
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
AbstractIn Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In …...
Improving Compositional Generalization with Self-Training for Data-to-Text Generation
AbstractData-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs). Such representations are compositional and it is costly to collect responses for all possible combinations of …...
Quality-Aware Decoding for Neural Machine Translation
AbstractDespite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the …...
Searching for COMETINHO: The Little Metric That Could
AbstractIn recent years, several neural fine-tuned machine translation evaluation metrics such as COMET and BLEURT have been proposed. These metrics achieve much higher correlations with human judgments than lexical overlap …...
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
AbstractAlthough neural-based machine translation evaluation metrics, such as COMET or BLEURT, have achieved strong correlations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered …...
BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
AbstractAutomatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained model-based metrics that …...
SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes
AbstractIs it possible to train a general metric for evaluating text generation quality without human-annotated ratings? Existing learned metrics either perform unsatisfactory across text generation tasks or require human ratings …...
Tagged End-to-End Simultaneous Speech Translation Training Using Simultaneous Interpretation Data
AbstractSimultaneous speech translation (SimulST) translates partial speech inputs incrementally. Although the monotonic correspondence between input and output is preferable for smaller latency, it is not the case for distant language …...