Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP) and Research Training Group AIPHES Department of Computer Science, Technische Universit¨at Darmstadt Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research www.ukp.tu-darmstadt.de Abstract In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tag- ging tasks that the seed value for the ran- dom number generator can result in statis- tically significant (p < 10−4) differences for state-of-the-art systems. For two re- cent systems for NER, we observe an ab- solute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived ei- ther as state-of-the-art or mediocre. Instead of publishing and reporting single perfor- mance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM- networks for five sequence tagging tasks, we present network architectures that pro- duce both superior performance as well as are more stable with respect to the remain- ing hyperparameters. The full experimen- tal results are published in (Reimers and Gurevych, 2017).1 The implementation of our network is publicly available.2 1 Introduction Large efforts are spent in our community on devel- oping new state-of-the-art approaches. To docu- ment that those approaches are better, they are ap- plied to unseen data and the obtained performance score is compared to previous approaches. In or- der to make results comparable, a provided split between train, development and test data is often 1https://arxiv.org/abs/1707.06799 2https://github.com/UKPLab/ emnlp2017-bilstm-cnn-crf used, for example from a former shared task. In recent years, deep neural networks were shown to achieve state-of-the-art performance for a wide range of NLP tasks, including many sequence tag- ging tasks (Ma and Hovy, 2016), dependency pars- ing (Andor et al., 2016), and machine translation (Wu et al., 2016). The training process for neural networks is highly non-deterministic as it usually depends on a random weight initialization, a ran- dom shuffling of the training data for each epoch, and repeatedly applying random dropout masks. The error function of a neural network is a highly non-convex function of the parameters with the potential for many distinct local minima (LeCun et al., 1998; Erhan et al., 2010). Depending on the seed value for the pseudo-random number genera- tor, the network will converge to a different local minimum. Our experiments show that these different local minima have vastly different characteristics on un- seen data. For the recent NER system by Ma and Hovy (2016) we observed that, depending on the random seed value, the performance on unseen data varies between 89.99% and 91.00% F1-score. The difference between the best and worst performance is statistically significant (p < 10−4) using a ran- domization test3. In conclusion, whether this newly developed approach is perceived …

Content truncated to 500 words