We Need to Talk About train-dev-test Splits

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4485–4494 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4485 We Need to Talk About train-dev-test Splits Rob van der Goot IT University of Copenhagen robv@itu.dk Abstract Standard train-dev-test splits used to bench- mark multiple models against each other are ubiquitously used in Natural Language Pro- cessing (NLP). In this setup, the train data is used for training the model, the development set for evaluating different versions of the pro- posed model(s) during development, and the test set to confirm the answers to the main re- search question(s). However, the introduction of neural networks in NLP has led to a differ- ent use of these standard splits; the develop- ment set is now often used for model selec- tion during the training procedure.Because of this, comparing multiple versions of the same model during development leads to overesti- mation on the development data. As an effect, people have started to compare an increasing amount of models on the test data, leading to faster overfitting and “expiration” of our test sets. We propose to use a tune-set when devel- oping neural network methods, which can be used for model picking so that comparing the different versions of a new model can safely be done on the development data.1 1 Dataset Splits in NLP 1.1 Current State In Natural Language Processing (NLP), a highly empirical field, it is common to benchmark mul- tiple models to each other on a standard dataset. However, since most current models are super- vised, and thus require labeled training data, the datasets have to be split. To ensure a fair com- parison, most datasets in NLP have standard splits. Most datasets consist of three splits (also visualized in Figure 1(a)): • train: Used for training models, in some se- tups this split can be omitted (zero-shot or unsupervised learning). 1Source code is available at https://bitbucket. org/robvanderg/tuneset Test phase Development phase (a) (b) (c) Figure 1: Overview of the use of data splits. red :test orange :dev green :train yellow :tune. a) standard splits for traditional machine learning models b) stan- dard splits as used for neural network models c) our proposed splits for neural network models. • development (also called validation/evalua- tion): Used to compare all different versions of the proposed model(s). Can also be used to get preliminary answers to the main research questions. • test: Used to confirm the final answer to the research question. One often raised worry is that if too many papers are written based on the same test-set, overfitting occurs, especially when only positive results are published (Scargle, 2000). It should be noted that we do not refer to overfitting of the models parame- ters, but on design decisions (hyperparameters etc.), in line with “bias from research design” as defined by Hovy and Prabhumoye (2021). This means that there is a bias towards methods that perform well on this specific set. We agree that this is a dan- ger. …

Content truncated to 500 words