Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages

Early Stopping Based on Unlabeled Samples in Text Classification HongSeok Choi1 Dongha Choi2 Hyunju Lee1,2,∗ 1School of Electrical Engineering and Computer Science, 2Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology, Gwangju 61005, South Korea {hongking9,hyunjulee}@gist.ac.kr, dongha528@gm.gist.ac.kr Abstract Early stopping, which is widely used to pre- vent overfitting, is generally based on a sep- arate validation set. However, in low re- source settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the re- duction in the number of samples by valida- tion split may result in insufficient samples for training. In this study, we propose an early stopping method that uses unlabeled samples. The proposed method is based on confidence and class distribution similarities. To further improve the performance, we present a cali- bration method to better estimate the class dis- tribution of the unlabeled samples. The pro- posed method is advantageous because it does not require a separate validation set and pro- vides a better stopping point by using a large unlabeled set. Extensive experiments are con- ducted on five text classification datasets and several stop-methods are compared. Our re- sults show that the proposed model even per- forms better than using an additional valida- tion set as well as the existing stop-methods, in both balanced and imbalanced data settings. Our code is available at https://github. com/DMCB-GIST/BUS-stop. 1 Introduction Early stopping, a form of regularization, is a widely used technique to prevent a model from over-fitting (Yao et al., 2007; Zhang et al., 2017). It is gener- ally based on a separate validation set (Goodfel- low et al., 2016). While monitoring the validation performance during training, the training process stops when the validation error starts to increase. Validation-based early stopping is advantageous be- cause it is easy to implement and can be interpreted directly (Prechelt, 1998). In a scenario where sufficient labeled data are available, the use of a validation set is generally pre- ferred (Goodfellow et al., 2016). However, when ∗Corresponding author only a few labeled data exist, a tradeoff problem is encountered (Kann et al., 2019; Choi and Lee, 2021). For example, although the usage of a rel- atively large validation set enables more reliable estimation, the number of samples for training be- comes insufficient. Conversely, if small fractions of the samples are assigned to the validation set, the stopping point becomes ambiguous because the small validation set is not representative enough. Early stopping is more important in a low re- source setting because the prediction accuracy fluc- tuates highly during training. Such high fluctua- tions render it challenging when to stop the model. One way to mitigate these fluctuations is to use suf- ficient training data. In this context, training all the available samples would be more effective, and for this purpose, an appropriate stopping point should be determined without validation split. However, this has not been extensively studied. Duvenaud et al. (2016) and Mahsereci et al. (2017) proposed gradient-based stop-methods and applied statisti- cal inference on …

Content truncated to 500 words