Repeat before Forgetting: Spaced Repetition for Efficient and Effective Training of Neural Networks

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2401–2410 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Repeat before Forgetting: Spaced Repetition for Efficient and Effective Training of Neural Networks Hadi Amiri, Timothy A. Miller, Guergana Savova Boston Children’s Hospital Informatics Program, Harvard Medical School {firstname.lastname}@childrens.harvard.edu Abstract We present a novel approach for training artificial neural networks. Our approach is inspired by broad evidence in psychology that shows human learners can learn effi- ciently and effectively by increasing inter- vals of time between subsequent reviews of previously learned materials (spaced repeti- tion). We investigate the analogy between training neural models and findings in psy- chology about human memory model and develop an efficient and effective algorithm to train neural models. The core part of our algorithm is a cognitively-motivated sched- uler according to which training instances and their “reviews” are spaced over time. Our algorithm uses only 34-50% of data per epoch, is 2.9-4.8 times faster than stan- dard training, and outperforms competing state-of-the-art baselines.1 1 Introduction Deep neural models are known to be computa- tionally expensive to train even with fast hard- ware (Sutskever et al., 2014; Wu et al., 2016). For example, it takes three weeks to train a deep neu- ral machine translation system on 100 Graphics Processing Units (GPUs) (Wu et al., 2016). Fur- thermore, a large amount of data is usually required to train effective neural models (Goodfellow et al., 2016; Hirschberg and Manning, 2015). Bengio et al. (2009) and Kumar et al. (2010) de- veloped training paradigms which are inspired by the learning principle that humans can learn more effectively when training starts with easier con- cepts and gradually proceeds with more difficult concepts. Since these approaches are motivated by 1Our code is available at scholar.harvard.edu/ hadi/RbF/ a “starting small” strategy they are called curricu- lum or self-paced learning. In this paper, we present a novel training paradigm which is inspired by the broad evidence in psychology that shows human ability to retain information improves with repeated exposure and exponentially decays with delay since last expo- sure (Cepeda et al., 2006; Averell and Heathcote, 2011). Spaced repetition was presented in psychol- ogy (Dempster, 1989) and forms the building block of many educational devices, including flashcards, in which small pieces of information are repeatedly presented to a learner on a schedule determined by a spaced repetition algorithm. Such algorithms show that human learners can learn efficiently and effectively by increasing intervals of time between subsequent reviews of previously learned materi- als (Dempster, 1989; Novikoff et al., 2012). We investigate the analogy between training neu- ral models and findings in psychology about human memory model and develop a spaced repetition al- gorithm (named Repeat before Forgetting, RbF) to efficiently and effectively train neural models. The core part of our algorithm is a scheduler that ensures a given neural network spends more time working on difficult training instances and less time on easier ones. Our scheduler is inspired by fac- …

Content truncated to 500 words