Language Model Augmented Relevance Score

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6677–6690 August 1–6, 2021. ©2021 Association for Computational Linguistics 6677 Language Model Augmented Relevance Score Ruibo Liu1 Jason Wei2 Soroush Vosoughi1 1Dartmouth College 2Google AI Language ruibo.liu.gr@dartmouth.edu jasonwei@google.com soroush.vosoughi@dartmouth.edu Abstract Although automated metrics are commonly used to evaluate NLG systems, they often cor- relate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which rely on 푛-gram matching. These newer methods, however, are still limited in that they do not consider the generation context, so they cannot properly re- ward generated text that is correct but deviates from the given reference. In this paper, we propose Language Model Augmented Relevance Score (MARS), a new context-aware metric for NLG evaluation. MARS leverages off-the-shelf language mod- els, guided by reinforcement learning, to cre- ate augmented references that consider both the generation context and available human references, which are then used as additional references to score generated text. Compared with seven existing metrics in three common NLG tasks, MARS not only achieves higher correlation with human reference judgements, but also differentiates well-formed candidates from adversarial samples to a larger degree. 1 Introduction Automated metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) are popular meth- ods for evaluating natural language generation (NLG) systems. Compared with human evalua- tion, they are cheaper and faster, and accordingly, they often serve as essential metrics for benchmark- ing the performance of NLG models (Novikova et al., 2017). Despite their widespread use, how- ever, these automated metrics often poorly correlate with ratings given by human judges, particularly for datasets in which only a single human reference exists (Gupta et al., 2019; Novikova et al., 2017). Moreover, these automated metrics only capture Context Exsiting Metrics: PPL / BLEU / ROUGE / BERT-Score / etc. Ours: MARS Human Reference Candidate LM LM Augmented References Figure 1: Existing metrics compare the candidate with the human reference but ignore context. MARS (our method) augments the human reference while consider- ing the context, which allows it to provide evaluation scores that correlate highly with human references. similarities between generated sentences and refer- ence candidates, crucially ignoring provided con- texts that are relevant for evaluating the answer in contextual NLG tasks, such as story generation, news summarization, and question-answering (Tao et al., 2018; Nema and Khapra, 2018). Table 1 shows a story generation1 example that exemplifies some weaknesses of several com- mon metrics. Perplexity (PPL) (Brown et al., 1992) successfully detects ungrammatical sen- tences, but it fails to distinguish legitimate novel continuations and copy-and-pasted ones. Rely- ing on surface-level 푛-gram matching, BLEU-1 and ROUGE-L2 cannot detect reordering effec- tively, and wrongly score the well-formed candi- date lower than its retrieval-based adversarial ex- ample. BERTScore (Zhang et al., 2019) leverages contextual embeddings from BERT (Devlin et al., 2019), thus mitigating the above challenges, but still does not …

Content truncated to 500 words