Towards More Fine-grained and Reliable NLP Performance Prediction

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3703–3714 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3703 Towards More Fine-grained and Reliable NLP Performance Prediction Zihuiwen Ye, Pengfei Liu, Jinlan Fu†, Graham Neubig Carnegie Mellon University, † Fudan University {zihuiwey,pliu3,gneubig}@andrew.cmu.edu, fujl16@fudan.edu.cn Abstract Performance prediction, the task of estimating a system’s performance without performing experiments, allows us to reduce the experi- mental burden caused by the combinatorial ex- plosion of different datasets, languages, tasks, and models. In this paper, we make two contri- butions to improving performance prediction for NLP tasks. First, we examine performance predictors not only for holistic measures of ac- curacy like F1 or BLEU, but also fine-grained performance measures such as accuracy over individual classes of examples. Second, we propose methods to understand the reliabil- ity of a performance prediction model from two angles: confidence intervals and calibra- tion. We perform an analysis of four types of NLP tasks, and both demonstrate the feasibil- ity of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future. We make our code publicly available: https: //github.com/neulab/Reliable-NLPPP 1 Introduction Performance prediction (P2) aims to predict a ma- chine learning system’s performance based on fea- tures of the underlying problem, dataset, or learn- ing algorithm. While this topic is still relatively unexplored in the NLP context, there are a few ex- amples of predicting performance as: (i) a func- tion of training or model parameters for determin- ing the number of training iterations (Kolachina et al., 2012) or value of hyperparameters (Rosen- feld et al., 2019) and identifying and terminat- ing bad training runs (Domhan et al., 2015). (ii) a function of dataset characteristics to illustrate which factors are significant predictors of sys- tem performance (Birch et al., 2008; Turchi et al., 2008), or find a subset of representative experi- ments to run in order to obtain plausible predic- tions (Xia et al., 2020). In this paper, we ask two (0, 1] (1, 2] (2, 3] (3, 6] Entity Length Interval 0.6 0.7 0.8 0.9 1.0 F1 3206 1631 192 49 Figure 1: Breakdown of performance over different entity lengths of an NER system. Actual F1 (gray point) is cal- culated from actual results while predicted F1 (red point) is estimated by a performance prediction model. Gray and red lines represent corresponding confidence intervals. Numbers in each bar indicate the number of test samples in each length bucket. research questions with respect to performance prediction: can we predict performance on a more fine-grained level, and can we quantify the relia- bility of performance predictions? With respect to the first contribution, previous P2 methods have almost entirely focused on pre- dicting holistic measures of accuracy such as en- tity F1 (Ratinov and Roth, 2009) or BLEU score (Papineni et al., 2002) over the entire dataset (§2.2). However, from a perspective of under- standing the workings of our models, work on model analysis has demonstrated …

Content truncated to 500 words