Easy Questions First? A Case Study on Curriculum Learning for Question Answering

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 453–463, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Easy Questions First? A Case Study on Curriculum Learning for Question Answering Mrinmaya Sachan Eric P. Xing School of Computer Science Carnegie Mellon University {mrinmays, epxing}@cs.cmu.edu Abstract Cognitive science researchers have em- phasized the importance of ordering a complex task into a sequence of easy to hard problems. Such an ordering provides an easier path to learning and increases the speed of acquisition of the task com- pared to conventional learning. Recent works in machine learning have explored a curriculum learning approach called self- paced learning which orders data samples on the easiness scale so that easy sam- ples can be introduced to the learning algo- rithm first and harder samples can be intro- duced successively. We introduce a num- ber of heuristics that improve upon self- paced learning. Then, we argue that incor- porating easy, yet, a diverse set of samples can further improve learning. We compare these curriculum learning proposals in the context of four non-convex models for QA and show that they lead to real improve- ments in each of them. 1 Introduction A key challenge in building an intelligent agent is in modeling the incrementality and the cumulative nature of human learning (Skinner, 1958; Peter- son, 2004; Krueger and Dayan, 2009). Children typically learn grade by grade, progressing from simple concepts to more complex ones. Given a complex set of concepts, it is often the case that some concepts are easier than others. Some con- cepts can even be prerequisite to learning other concepts. Hence, evolving a useful curriculum where easy concepts are presented first and more complex concepts are gradually introduced can be beneficial for learning. We explore methods for learning a curriculum in the context of non-convex models for question answering. Curriculum learning (CL) (Bengio et al., 2009) and self-paced learning (SPL) (Kumar et al., 2010) have been recently introduced in ma- chine learning literature. However, their useful- ness in the context of NLP tasks such as QA has not been studied so far. The main challenge in learning a curriculum is that it requires the identifi- cation of easy and hard concepts in the given train- ing dataset. However, in real-world applications, such a ranking of training samples is difficult to obtain. Furthermore, a human judgement of ‘eas- iness’ of a task might not correlate with what is easy for the algorithm in the feature and hypothe- sis space employed for the given application. SPL combines the selection of the curriculum and the learning task in a single objective. The easiness of a question in self-paced learning is defined by its local loss. We propose and study other heuristics that define a measure of easiness and learn the cur- riculum by selecting samples using this measure. These heuristics are similar to those used in active learning, but with one key difference. In curricu- lum learning, all the training examples …

Content truncated to 500 words