Curriculum Learning for Graph Neural Networks: A Multiview Competence-based Approach

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7036–7051 July 9-14, 2023 ©2023 Association for Computational Linguistics Curriculum Learning for Graph Neural Networks: A Multiview Competence-based Approach Nidhi Vakil Department of Computer Science University of Massachusetts Lowell nvakil@cs.uml.edu Hadi Amiri Department of Computer Science University of Massachusetts Lowell hadi@cs.uml.edu Abstract A curriculum is a planned sequence of learning materials and an effective one can make learn- ing efficient and effective for both humans and machines. Recent studies developed effective data-driven curriculum learning approaches for training graph neural networks in language ap- plications. However, existing curriculum learn- ing approaches often employ a single criterion of difficulty in their training paradigms. In this paper, we propose a new perspective on curricu- lum learning by introducing a novel approach that builds on graph complexity formalisms (as difficulty criteria) and model competence dur- ing training. The model consists of a schedul- ing scheme which derives effective curricula by accounting for different views of sample difficulty and model competence during train- ing. The proposed solution advances existing research in curriculum learning for graph neu- ral networks with the ability to incorporate a fine-grained spectrum of graph difficulty crite- ria in their training paradigms. Experimental results on real-world link prediction and node classification tasks illustrate the effectiveness of the proposed approach.1 1 Introduction Graph Neural Networks (GNNs) are generally trained using stochastic gradient descent (SGD), where the standard approach is to iteratively use the entire training data to optimize model’s objective until convergence. Curriculum learning techniques improve this training process by scheduling exam- ples for training, e.g., by gradually learning from easier examples before training with harder ones. Such curricula can be predefined by humans (Ben- gio and LeCun, 2007; Bengio et al., 2009) or dy- namically derived from data during training (Jiang et al., 2018; Castells et al., 2020). Curriculum learning for graph neural networks is an emerging area of research. Recently, Chu et al. 1Code, data splits and guidelines are available at https: //clu.cs.uml.edu/tools.html. (2021) employed a traditional curriculum learning approach introduced in (Bengio et al., 2009) to im- prove negative sampling for graph classification. Wang et al. (2021) proposed to estimate the diffi- culty of graph entities–nodes, edges or subgraphs– based on the intra- and inter-class distributions of their embeddings in supervised settings, and devel- oped a smooth-step function to gradually introduce harder examples to GNNs during training. Vakil and Amiri (2022) developed a loss-based curricu- lum learning approach that dynamically adjusts the difficulty boundaries of training samples based on their sample-level loss trajectories obtained from recent training dynamics of GNN models. To the best of our knowledge, existing curricu- lum learning approaches often employ a single criterion of difficulty in their curriculum learn- ing framework, e.g., prediction loss (Wu et al., 2021), consistency in prediction loss (Xu et al., 2020), moving average of loss (Zhou et al., 2020) or transformations of loss (Vakil and Amiri, 2022). We address this gap by developing a …

Content truncated to 500 words