Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3342–3349, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3342 Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set Katharina Kann, Kyunghyun Cho and Samuel R. Bowman New York University, USA {kann, kyunghyun.cho, bowman}@nyu.edu Abstract Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effec- tive. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting in- fluence results as compared to a more realis- tic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or under- estimation of performance? We repeat mul- tiple experiments from recent work on neural models for low-resource NLP and compare re- sults for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic ex- perimental setups in the publication of low- resource NLP research results. 1 Introduction Parametric machine learning models are fre- quently trained by minimizing the loss on the training set T , LT (θ) = � x∈T l (θ, x) , (1) for model parameters θ and a predefined loss func- tion l. Gradient-based optimizers minimize this loss L(θ) by updating θ in the direction of the gra- dient ∇L(θ). A low loss characterizes a model which makes accurate predictions for examples in T . However, since T is finite, overfitting the train- ing set might lead to poor generalization perfor- mance. One way to avoid fitting Equation 1 too # train # dev ES Bollmann et al. (2018) 5k 12k-46k Yes Kann et al. (2018) 400-700 100-200 Yes Makarov and Clematide (2018) 100 1k Yes Sharma et al. (2018) 100 100 Yes Schulz et al. (2018) 1k-21k 9k N/A Upadhyay et al. (2018) 500 1k Yes Table 1: Number of examples used for training and development in recent low-resource NLP experiments; ES=early stopping on the development set. Experi- ments from papers in bold will be revisited here. closely is early stopping: a separate development or validation set is used to end training as soon as the loss on the development set LD(θ) starts increasing or model performance on the develop- ment set D starts decreasing. The best set of pa- rameters θ is used in the final model. This works well when large amounts of data are available to create training, development and test splits. Recently, however, with the success of pre- training (Peters et al., 2018; Devlin et al., 2019) and multi-task learning (Caruana, 1997; Ruder, 2017; Wang et al., 2019) approaches, …

Content truncated to 500 words