Dynamic Human Evaluation for Relative Model Comparisons

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5946–5955 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5946 Dynamic Human Evaluation for Relative Model Comparisons Th´orhildur Thorleiksd´ottir1, Cedric Renggli1, Nora Hollenstein2, Ce Zhang1 1ETH Z¨urich, tthorleiks@gmail.com, {cedric.renggli, ce.zhang}@inf.ethz.ch 2University of Copenhagen, nora.hollenstein@hum.ku.dk Abstract Collecting human judgements is currently the most reliable evaluation method for natural language generation systems. Automatic metrics have reported flaws when applied to measure quality aspects of generated text and have been shown to correlate poorly with human judgements. However, human evaluation is time and cost-intensive, and we lack consensus on designing and conducting human evaluation experiments. Thus there is a need for streamlined approaches for efficient collection of human judgements when evaluating natural language generation systems. Therefore, we present a dynamic approach to measure the required number of human annotations when evaluating generated outputs in relative comparison settings. We propose an agent-based framework of human evaluation to assess multiple labelling strategies and methods to decide the better model in a simulation and a crowdsourcing case study. The main results indicate that a decision about the superior model can be made with high probability across different labelling strategies, where assigning a single random worker per task requires the least overall labelling effort and thus the least cost. Keywords: Human Evaluation, Crowdsourcing, Natural Language Generation, Relative Model Comparison 1. Introduction Human evaluation is regarded as the primary evaluation metric for natural language generation (NLG) systems due to the lack of automatic metrics that successfully encode quality aspects of generated text (Chaganty et al., 2018; Celikyilmaz et al., 2020). Still, evaluating systems with human judgements comes with several challenges. Human evaluations are expensive and time- consuming and often demand high-quality judgements (e.g., hiring experts, training non-experts) (Celikyilmaz et al., 2020). A fixed number of annotations can lead to evaluation experiments that are likely to be statis- tically underpowered to detect the true effects of the corresponding model (Card et al., 2020). Little con- sensus exists on how to design, conduct, and report human evaluations (Howcroft et al., 2020), which has a significant impact on the reliability of the collected human judgements (Novikova et al., 2018; Santhanam and Shaikh, 2019) and makes it difficult to compare progress across different systems. When developing NLG systems, automatic metrics are generally applied to track progress (Celikyilmaz et al., 2020) despite the fact that common metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) have reported flaws and correlate poorly with hu- man judgements (Gehrmann et al., 2021; Novikova et al., 2017). These limitations highlight the importance of human evaluation during the development of NLG models to capture the progress of necessary quality as- pects of the generated text and to support further im- provements for automatic metrics. The ultimate goal is to support the integration of a safe deployment pipeline for NLG models. Once a model with new improve- ments is estimated to be production-ready, it is eval- uated against …

Content truncated to 500 words