Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465–476 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 465 Dice Loss for Data-imbalanced NLP Tasks Xiaoya Li♣, Xiaofei Sun♣, Yuxian Meng♣, Junjun Liang♣, Fei Wu♠ and Jiwei Li♠♣ ♠ Department of Computer Science and Technology, Zhejiang University ♣ Shannon.AI {xiaoya li, xiaofei sun, yuxian meng, jiwei li}@shannonai.com, wufei@cs.zju.edu.cn Abstract Many NLP tasks such as tagging and ma- chine reading comprehension (MRC) are faced with the severe data imbalance issue: nega- tive examples significantly outnumber positive ones, and the huge number of easy-negative examples overwhelms training. The most commonly used cross entropy criteria is ac- tually accuracy-oriented, which creates a dis- crepancy between training and test. At train- ing time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive ex- amples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy ob- jective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen–Dice coefficient (Sorensen, 1948) or Tversky index (Tversky, 1977), which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To fur- ther alleviate the dominating influence from easy-negative examples in training, we pro- pose to associate training examples with dy- namically adjusted weights to deemphasize easy-negative examples. Experimental results show that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we ob- serve significant performance boosts over a wide range of data imbalanced NLP tasks. No- tably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task, and competitive or even better results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named en- tity recognition task along with the machine reading comprehension and paraphrase iden- tification tasks. The code can be found at https://github.com/ShannonAI/ dice_loss_for_NLP. Task # neg # pos ratio CoNLL03 NER 170K 34K 4.98 OntoNotes5.0 NER 1.96M 239K 8.18 SQuAD 1.1 (Rajpurkar et al., 2016) 10.3M 175K 55.9 SQuAD 2.0 (Rajpurkar et al., 2018) 15.4M 188K 82.0 QUOREF (Dasigi et al., 2019) 6.52M 38.6K 169 Table 1: Number of positive and negative examples and their ratios for different data-imbalanced NLP tasks. 1 Introduction Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine read- ing comprehension. Table 1 gives concrete exam- ples: for the Named Entity Recognition (NER) task (Sang and De Meulder, 2003; Nadeau and Sekine, 2007), most tokens are backgrounds with tagging class O. Specifically, the number of to- kens with tagging class O is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5.0 dataset; Data- imbalanced issue is more severe for MRC tasks (Rajpurkar et al., 2016; Nguyen et al., 2016; Ra- jpurkar et al., 2018; Koˇcisk`y et al., 2018; Dasigi et al., 2019) …

Content truncated to 500 words