Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 130–139, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning Yulia Tsvetkov♠ Manaal Faruqui♠ Wang Ling♣ Brian MacWhinney♠ Chris Dyer♣♠ ♠Carnegie Mellon University ♣Google DeepMind {ytsvetko,mfaruqui,cdyer}@cs.cmu.edu, lingwang@google.com, macw@cmu.edu Abstract We use Bayesian optimization to learn curricula for word representation learning, optimizing performance on downstream tasks that depend on the learned represen- tations as features. The curricula are mod- eled by a linear ranking function which is the scalar product of a learned weight vec- tor and an engineered feature vector that characterizes the different aspects of the complexity of each instance in the training corpus. We show that learning the curricu- lum improves performance on a variety of downstream tasks over random orders and in comparison to the natural corpus order. 1 Introduction It is well established that in language acquisition, there are robust patterns in the order by which phenomena are acquired. For example, prototypi- cal concepts are acquired earlier; concrete words tend to be learned before abstract ones (Rosch, 1978). The acquisition of lexical knowledge in artificial systems proceeds differently. In gen- eral, models will improve during the course of pa- rameter learning, but the time course of acquisi- tion is not generally studied beyond generaliza- tion error as a function of training time or data size. We revisit this issue of choosing the order of learning—curriculum learning—framing it as an optimization problem so that a rich array of factors—including nuanced measures of difficulty, as well as prototypicality and diversity—can be exploited. Prior research focusing on curriculum strate- gies in NLP is scarce, and has conventionally been following a paradigm of “starting small” (Elman, 1993), i.e., initializing the learner with “simple” examples first, and then gradually increasing data complexity (Bengio et al., 2009; Spitkovsky et al., 2010). In language modeling, this prefer- ence for increasing complexity has been realized by curricula that increase the entropy of training data by growing the size of the training vocabu- lary from frequent to less frequent words (Ben- gio et al., 2009). In unsupervised grammar in- duction, an effective curriculum comes from in- creasing length of training sentences as training progresses (Spitkovsky et al., 2010). These case studies have demonstrated that carefully designed curricula can lead to better results. However, they have relied on heuristics in selecting curricula or have followed the intuitions of human and animal learning (Kail, 1990; Skinner, 1938). Had differ- ent heuristics been chosen, the results would have been different. In this paper, we use curriculum learning to create improved word representations. However, rather than testing a small number of curricula, we search for an optimal curriculum us- ing Bayesian optimization. A curriculum is de- fined to be the ordering of the training instances, in our case it is the ordering of paragraphs in which the representation learning model reads the cor- pus. We use a linear ranking function to conduct a systematic …

Content truncated to 500 words