K-hop neighbourhood regularization for few-shot learning on graphs: A case study of text classification

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1187–1200 May 2-6, 2023 ©2023 Association for Computational Linguistics FewShotTextGCN: K-hop neighborhood regularization for few-shot learning on graphs Niels van der Heijden♣♢ Ekaterina Shutova♣ Helen Yannakoudakis♠ ♣ILLC, University of Amsterdam, the Netherlands ♠Dept. of Informatics, King’s College London, United Kingdom ♢Deloitte Risk Advisory, Amsterdam, the Netherlands n.vanderheijden@uva.nl, e.shutova@uva.nl, helen.yannakoudakis@kcl.ac.uk Abstract We present FewShotTextGCN, a novel method designed to effectively utilize the properties of word–document graphs for improved learn- ing in low-resource settings. We introduce K- hop Neighborhood Regularization, a regular- izer for heterogeneous graphs, and show that it stabilizes and improves learning when only a few training samples are available. We fur- thermore propose a simplification in the graph- construction method, which results in a graph that is ∼7 times less dense and yields better performance in low-resource settings while per- forming on-par with the state of the art in high- resource settings. Finally, we introduce a new variant of Adaptive Pseudo-Labeling tailored for word–document graphs. When using as lit- tle as 20 samples for training, we outperform a strong TextGCN baseline with 17% in absolute accuracy on average over eight languages. We demonstrate that our method can be applied to document classification without any language model pretraining on a wide range of typolog- ically diverse languages while performing on par with large pretrained language models. 1 Introduction Text classification, a key task in natural language processing (NLP), has many real-world applica- tions, including toxic comment identification, news categorization, spam detection and opinion mining. One popular approach to this problem relies on large-scale pretraining of Transformer models (De- vlin et al., 2018; Conneau et al., 2019; Raffel et al., 2020), which have shown to be able to approach or even surpass human performance on many natural language understanding (NLU) benchmarks (Ra- jpurkar et al., 2016; Wang et al., 2019; Liang et al., 2020). While these results are impressive for the languages on which models are pretrained, perfor- mance tends to deteriorate on languages where no or little data is available (Chau and Smith, 2021; van der Heijden et al., 2020). In practice, this means that these models are effective on a set of approximately 100 out of the 7000+ spoken lan- guages in the world. Next to the requirement for vast amounts of data for pretraining, Transformer language models tend to be impractically large in terms of their number of parameters and have a high environmental footprint (Strubell et al., 2019). Recently, Graph Neural Networks (GNNs) have shown to be effective for text classification in both transductive (Yao et al., 2019; Liu et al., 2020; Lin et al., 2021) and inductive (Nikolentzos et al., 2020; Ding et al., 2020) learning settings – with promising results in both high- and low-resource settings. Particularly in the transductive setting, the authors of TextGCN (Yao et al., 2019) show that Graph Convolutional Networks (GCNs) (Kipf and Welling, 2016) can outperform state-of-the-art methods for document classification on English datasets …

Content truncated to 500 words