NoiseQA: Challenge Set Evaluation for User-Centric Question Answering

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 326–346 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 326 Evaluating the Evaluation of Diversity in Natural Language Generation Guy Tevet1,2 Jonathan Berant1,3 1School of Computer Science, Tel-Aviv University 2Department of Electrical Engineering, Tel-Aviv University 3Allen Institute for AI {guytevet@mail,joberant@cs}.tau.ac.il Abstract Despite growing interest in natural language generation (NLG) models that produce di- verse outputs, there is currently no principled method for evaluating the diversity of an NLG system. In this work, we propose a framework for evaluating diversity metrics. The frame- work measures the correlation between a pro- posed diversity metric and a diversity parame- ter, a single parameter that controls some as- pect of diversity in generated text. For ex- ample, a diversity parameter might be a bi- nary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity. We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans sub- stantially outperform automatic metrics in es- timating content diversity, and (c) demonstrat- ing that existing methods for controlling diver- sity by tuning a “decoding parameter” mostly affect form but not meaning. Our framework can advance the understanding of different di- versity metrics, an essential step on the road towards better NLG systems. 1 Introduction An important desideratum of natural language generation (NLG) systems is to produce outputs that are not only correct, but also diverse. For ex- ample, a dialog system (Adiwardana et al., 2020) should permit many responses for the prompt “How are you today?”. Similarly, we expect di- verse responses in tasks such as story generation (Li et al., 2018), question generation (Pan et al., 2019) and question answering (Fan et al., 2019). Despite growing effort to produce more diverse models (Li et al., 2016c,a; Holtzman et al., 2019; Du and Black, 2019), there is no standard eval- uation metric for measuring diversity. Thus, dif- ferent papers evaluate diversity differently (if at 0.6 0.8 1.0 Metric Values distinct-n (averaged) Set A Set B 2 3 4 Metric Values absHDS Question: So what did I miss in the first 20 minutes? Set A • Pretty much everything. • Nothing, really. • You won’t believe what happened! • Why do you even care? • What were you doing that was more important than this? Set B • Not much. • It was pretty dull. • Blah, you didn’t miss anything. • Not anything that important. • Very little, it was uneventful. Figure 1: Diversity metric evaluation: we show two sets of responses to the same question, generated by crowdsourcing workers. While both sets are diverse in terms of form, only set A is diverse in terms of content. Each graph presents the distribution over a diversity metric for sets with high content diversity (blue) and low content diversity (orange). Distri- butions are approximated over 200 sets. We observe that the human score …

Content truncated to 500 words