A Comparative Study on Regularization Strategies for Embedding-based Neural Networks

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2106–2111, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Comparative Study on Regularization Strategies for Embedding-based Neural Networks Hao Peng∗, 1 Lili Mou,∗1 Ge Li,†1 Yunchuan Chen,2 Yangyang Lu,1 Zhi Jin1 1Software Institute, Peking University, 100871, P. R. China {penghao.pku, doublepower.mou}@gmail.com,{lige, luyy11, zhijin}@sei.pku.edu.cn 2University of Chinese Academy of Sciences, chenyunchuan11@mails.ucas.ac.cn Abstract This paper aims to compare different reg- ularization strategies to address a com- mon phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neu- ral models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re- embedding words, and dropout. We also emphasized on incremental hyperparame- ter tuning, and combining different regu- larizations. The results provide a picture on tuning hyperparameters for neural NLP models. 1 Introduction Neural networks have exhibited considerable po- tential in various fields (Krizhevsky et al., 2012; Graves et al., 2013). In early years on neural NLP research, neural networks were used in lan- guage modeling (Bengio et al., 2003; Morin and Bengio, 2005; Mnih and Hinton, 2009); recently, they have been applied to various supervised tasks, such as named entity recognition (Collobert and Weston, 2008), sentiment analysis (Socher et al., 2011; Mou et al., 2015), relation classification (Zeng et al., 2014; Xu et al., 2015), etc. In the field of NLP, neural networks are typically combined with word embeddings, which are usually first pre- trained by unsupervised algorithms like Mikolov et al. (2013); then they are fed forward to standard neural models, fine-tuned during supervised learn- ing. However, embedding-based neural networks usually suffer from severe overfitting because of the high dimensionality of parameters. ∗Equal contribution. †Corresponding author. A curious question is whether we can regular- ize embedding-based NLP neural models to im- prove generalization. Although existing and newly proposed regularization methods might alleviate the problem, their inherent performance in neural NLP models is not clear: the use of embeddings is sparse; the behaviors may be different from those in other scenarios like image recognition. Further, selecting hyperparameters to pursue the best performance by validation is extremely time- consuming, as suggested in Collobert et al. (2011). Therefore, new studies are needed to provide a more complete picture regarding regularization for neural natural language processing. Specifically, we focus on the following research questions in this paper. RQ 1: How do different regularization strategies typically behave in embedding-based neural networks? RQ 2: Can regularization coefficients be tuned in- crementally during training so as to ease the burden of hyperparameter tuning? RQ 3: What is the effect of combining different regularization strategies? In this paper, we systematically and quan- titatively compared four different regularization strategies, namely penalizing weights, penalizing embeddings, newly proposed word re-embedding (Labutov and Lipson, 2013), and dropout (Srivas- tava et al., 2014). We analyzed these regulariza- tion methods by two widely studied models and tasks. We also emphasized on incremental hyper- …

Content truncated to 500 words