Hierarchical Attention Prototypical Networks for Few-Shot Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 476–485, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 476 Hierarchical Attention Prototypical Networks for Few-Shot Text Classification Shengli Sun1∗ Qingfeng Sun1∗ Kevin Zhou2 † Tengchao Lv1 1Peking University 2Microsoft slsun@ss.pku.edu.cn {sunqingfeng, lvtengchao}@pku.edu.cn kezhou@microsoft.com Abstract Most of the current effective methods for text classification task are based on large-scale la- beled data and a great number of parame- ters, but when the supervised training data are few and difficult to be collected, these mod- els are not available. In this paper, we pro- pose a hierarchical attention prototypical net- works (HAPN) for few-shot text classifica- tion. We design the feature level, word level, and instance level multi cross attention for our model to enhance the expressive ability of se- mantic space. We verify the effectiveness of our model on two standard benchmark few- shot text classification datasets - FewRel and CSID, and achieve the state-of-the-art perfor- mance. The visualization of hierarchical atten- tion layers illustrates that our model can cap- ture more important features, words, and in- stances separately. In addition, our attention mechanism increases support set augmentabil- ity and accelerates convergence speed in the training stage. 1 Introduction The dominant text classification models in deep learning (Kim, 2014; Zhang et al., 2015a; Yang et al., 2016; Wang et al., 2018) require a consider- able amount of labeled data to learn a large num- ber of parameters. However, such methods may have difficulty in learning the semantic space in the case that only few data are available. Few-shot learning has became an effective approach to solve this challenge, it can train a neural network with a few parameters using few data but achieve good performance. A typical example of this approach is prototypical networks (Snell et al., 2017), which averages the vector of few support instances as the class prototype and computes distance between target query and each prototype, then classify the query to the nearest prototype’s class. However, ∗ They contribute equally to this work. † Corresponding author. prototypical networks is rough and does not con- sider the adverse effects of various noises in the data, which weakens the discrimination and ex- pressiveness of the prototype. In this paper, we propose a hierarchical atten- tion prototypical networks for few-shot text clas- sification by using attention mechanism in three levels. For feature level attention, we use convo- lutional neural networks to get the feature scores which is different for various classes. For word level attention, we adopt an attention mechanism to learn the importance of each word hidden state in an instance. For instance level multi cross atten- tion, with the help of multi cross attention between support set and target query, we can determine the importance of different instances in the same class and enable the model to get a more discriminative prototype of each class. In the actual scenario, we apply HAPN …

Content truncated to 500 words