Predicting Performance for Natural Language Processing Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8625–8646 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8625 Predicting Performance for Natural Language Processing Tasks Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, Graham Neubig Language Technologies Institute, Carnegie Mellon University {mengzhox,aanastas,yiming,gneubig}@cs.cmu.edu ruochenx@gmail.com Abstract Given the complexity of combinations of tasks, languages, and domains in natural lan- guage processing (NLP) research, it is com- putationally prohibitive to exhaustively test newly proposed models on each possible ex- perimental setting. In this work, we attempt to explore the possibility of gaining plausi- ble judgments of how well an NLP model can perform under an experimental setting, with- out actually training or testing the model. To do so, we build regression models to predict the evaluation score of an NLP experiment given the experimental settings as input. Ex- perimenting on 9 different NLP tasks, we find that our predictors can produce meaningful predictions over unseen languages and differ- ent modeling architectures, outperforming rea- sonable baselines as well as human experts. Going further, we outline how our predictor can be used to find a small subset of represen- tative experiments that should be run in order to obtain plausible predictions for all other ex- perimental settings.1 1 Introduction Natural language processing (NLP) is an extraor- dinarily vast field, with a wide variety of models being applied to a multitude of tasks across a plen- itude of domains and languages. In order to mea- sure progress in all these scenarios, it is necessary to compare performance on test datasets represent- ing each scenario. However, the cross-product of tasks, languages, and domains creates an explo- sion of potential application scenarios, and it is in- feasible to collect high-quality test sets for each. In addition, even for tasks where we do have a wide variety of test data, e.g. for well-resourced tasks such as machine translation (MT), it is still 1Code, data and logs are publicly available at https: //github.com/xiamengzhou/NLPerf. computationally prohibitive as well as not environ- mentally friendly (Strubell et al., 2019) to build and test on systems for all languages or domains we are interested in. Because of this, the common practice is to test new methods on a small num- ber of languages or domains, often semi-arbitrarily chosen based on previous work or the experi- menters’ intuition. As a result, this practice impedes the NLP community from gaining a comprehensive under- standing of newly-proposed models. Table 1 il- lustrates this fact with an example from bilingual lexicon induction, a task that aims to find word translation pairs from cross-lingual word embed- dings. As vividly displayed in Table 1, almost all the works report evaluation results on a differ- ent subset of language pairs. Evaluating only on a small subset raises concerns about making infer- ences when comparing the merits of these meth- ods: there is no guarantee that performance on English–Spanish (EN–ES, the only common evalu- ation dataset) is representative of the expected per- formance of the …

Content truncated to 500 words