Show Your Work: Improved Reporting of Experimental Results

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2185–2194, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2185 Show Your Work: Improved Reporting of Experimental Results Jesse Dodge♣ Suchin Gururangan♦ Dallas Card♥ Roy Schwartz♠♦ Noah A. Smith♠♦ ♣Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA ♥Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA ♠Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA {jessed,dcard}@cs.cmu.edu {suching,roys,noah}@allenai.org Abstract Research in natural language processing pro- ceeds, in part, by demonstrating that new mod- els achieve superior performance (e.g., accu- racy) on held-out test data, compared to pre- vious results. In this paper, we demonstrate that test-set performance scores alone are in- sufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially per- formance on validation data obtained during model development. We present a novel tech- nique for doing so: expected validation per- formance of the best-found model as a func- tion of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where au- thors would have reached a different conclu- sion if they had used more (or less) compu- tation. Our approach also allows us to esti- mate the amount of computation required to obtain a given accuracy; applying it to sev- eral recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparison, and provide code to allow researchers to use our technique.1 1 Introduction In NLP and machine learning, improved perfor- mance on held-out test data is typically used as an indication of the superiority of one method over others. But, as the field grows, there is an increasing gap between the large computa- tional budgets used for some high-profile exper- iments and the budgets used in most other work (Schwartz et al., 2019). This hinders meaning- ful comparison between experiments, as improve- ments in performance can, in some cases, be ob- 1https://github.com/allenai/allentune (Budget, [accuracy]) Budget that favors LR Budget that favors CNN 39.8 32.0 38.8 31.1 39.5 … LR val. accuracy CNN val. accuracy 38.9 26.1 26.4 40.5 36.1 … current practice: max max report corresponding test-set accuracies Hyperparameter assignments Figure 1: Current practice when compraing NLP mod- els is to train multiple instantiations of each, choose the best model of each type based on validation per- formance, and compare their performance on test data (inner box). Under this setup, (assuming test-set re- sults are similar to validation), one would conclude from the results above (hyperparameter search for two models on the 5-way SST classification task) that the CNN outperforms Logistic Regression (LR). In our proposed evaluation framework, we instead encourage practitioners to consider the expected validation …

Content truncated to 500 words