DynaSent: A Dynamic Benchmark for Sentiment Analysis

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2388–2404 August 1–6, 2021. ©2021 Association for Computational Linguistics 2388 DynaSent: A Dynamic Benchmark for Sentiment Analysis Christopher Potts∗ Stanford University cgpotts@stanford.edu Zhengxuan Wu∗ Stanford University wuzhengx@stanford.edu Atticus Geiger Stanford University atticusg@stanford.edu Douwe Kiela Facebook AI Research dkiela@fb.com Abstract We introduce DynaSent (‘Dynamic Senti- ment’), a new English-language benchmark task for ternary (positive/negative/neutral) sen- timent analysis. DynaSent combines natu- rally occurring sentences with sentences cre- ated using the open-source Dynabench Plat- form, which facilities human-and-model-in- the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance perfor- mance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent ver- sion 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent’s Neutral category is more coherent than the compara- ble category in other benchmarks, and we mo- tivate training models from scratch for each round over successive fine-tuning. 1 Introduction Sentiment analysis is an early success story for NLP, in both a technical and an industrial sense. It has, however, entered into a more challenging phase for research and technology development: while present-day models achieve outstanding re- sults on all available benchmark tasks, they still fall short when deployed as part of real-world sys- tems (Burn-Murdoch, 2013; Grimes, 2014, 2017; Gossett, 2020) and display a range of clear short- comings (Kiritchenko and Mohammad, 2018; Han- wen Shen et al., 2018; Wallace et al., 2019; Tsai et al., 2019; Jin et al., 2019; Zhang et al., 2020). In this paper, we seek to address the gap between benchmark results and actual utility by introduc- ∗Equal contribution. Model 0 RoBERTa fine-tuned on sentiment benchmarks Model 0 used to find challenging naturally occurring sentences Human validation Round 1 Dataset Model 1 RoBERTa fine-tuned on sentiment benchmarks + Round 1 Dataset Dynabench used to crowdsource sentences that fool Model 1 Human validation Round 2 Dataset Figure 1: The DynaSent dataset creation process. The human validation task is the same for both rounds; five responses are obtained for each sentence. On Dyn- abench, we explore conditions with and without prompt sentences that workers can edit to achieve their goal. ing version 1 of the DynaSent dataset for English- language ternary (positive/negative/neutral) senti- ment analysis.1 DynaSent is intended to be a dy- namic benchmark that expands in response to new models, new modeling goals, and new adversarial attacks. We present the first two rounds here and motivate some specific data collection and mod- eling choices, and we propose that, when future models solve these rounds, we use those models to create additional DynaSent rounds. This is an instance of “the ‘moving …

Content truncated to 500 words