Deep Dominance - How to Properly Compare Deep Neural Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2773–2785 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2773 Deep Dominance - How to Properly Compare Deep Neural Models Rotem Dror Segev Shlomov Faculty of Industrial Engineering and Management, Technion, IIT rtmdrr|segevs|roiri@technion.ac.il Roi Reichart Abstract Comparing between Deep Neural Network (DNN) models based on their performance on unseen data is crucial for the progress of the NLP field. However, these models have a large number of hyper-parameters and, be- ing non-convex, their convergence point de- pends on the random values chosen at ini- tialization and during training. Proper DNN comparison hence requires a comparison be- tween their empirical score distributions on unseen data, rather than between single eval- uation scores as is standard for more simple, convex models. In this paper, we propose to adapt to this problem a recently proposed test for the Almost Stochastic Dominance relation between two distributions. We define the cri- teria for a high quality comparison method between DNNs, and show, both theoretically and through analysis of extensive experimen- tal results with leading DNN models for se- quence tagging tasks, that the proposed test meets all criteria while previously proposed methods fail to do so. We hope the test we propose here will set a new working practice in the NLP community.1 1 Introduction A large portion of the research activity in Natural Language Processing (NLP) is devoted to the de- velopment of new algorithms for existing or new tasks. To evaluate the quality of a new method, its performance on unseen datasets is compared to the performance of existing methods. The progress of the field hence crucially depends on our ability to draw conclusions from such comparisons. In the past, most supervised NLP models have been linear (or log-linear), convex and relatively simple (e.g. (Toutanova et al., 2003; Finkel et al., 2008; Ritter et al., 2011)). Hence, their training 1Our code is available at: https://github.com/ rtmdrr/deepComparison was deterministic and the number of configura- tions a model could have was rather small – deci- sions about model design were usually limited to feature selection and the selection of one of a few loss functions. Consequently, when one model performed better than another on unseen data it was safe to argue that the winning model was gen- erally better, especially when the results were sta- tistically significant (Dror et al., 2018), and when the effect of multiple hypothesis testing was taken into account in cases of evaluation with multiple datasets (Dror et al., 2017). With the recent emergence of Deep Neural Net- works (DNNs), data-driven performance compar- ison has become much more complicated. While models such as LSTM (Hochreiter and Schmid- huber, 1997), Bi-LSTM (Schuster and Paliwal, 1997) and the transformer (Vaswani et al., 2017) improved the state-of-the-art in many NLP tasks (e.g. (Dozat and Manning, 2017; Hershcovich et al., 2017; Yadav and Bethard, 2018)), it is much more difficult to compare the performance of algorithms that …

Content truncated to 500 words