Beyond Text: Incorporating Metadata and Label Structure for Multi-Label Document Classification using Heterogeneous Graphs

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3162–3171 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3162 Beyond Text: Incorporating Metadata and Label Structure for Multi-Label Document Classification using Heterogeneous Graphs Chenchen Ye1, Linhai Zhang1, Deyu Zhou1⇤, Yulan He2 3, Wu jie4 1School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China 2University of Warwick, 3Alan Turing Institute, UK 4Huawei Software Technologies Co., Ltd. {chenchenye,lzhang472,d.zhou}@seu.edu.cn yulan.he@warwick.ac.uk, wujie79@huawei.com Abstract Multi-label document classification, associat- ing one document instance with a set of rel- evant labels, is attracting more and more re- search attention. Existing methods explore the incorporation of information beyond text, such as document metadata or label structure. These approaches however either simply uti- lize the semantic information of metadata or employ the predefined parent-child label hi- erarchy, ignoring the heterogeneous graphical structures of metadata and labels, which we be- lieve are crucial for accurate multi-label doc- ument classification. Therefore, in this pa- per, we propose a novel neural network based approach for multi-label document classifica- tion, in which two heterogeneous graphs are constructed and learned using heterogeneous graph transformers. One is metadata hetero- geneous graph, which models various types of metadata and their topological relations. The other is label heterogeneous graph, which is constructed based on both the labels’ hierarchy and their statistical dependencies. Experimen- tal results on two benchmark datasets show the proposed approach outperforms several state- of-the-art baselines. 1 Introduction With the rapid growth of scientific documents, it is difficult to track related literature manually. For example, there are more than 200,000 publications related to COVID-19 by April 2021. Therefore, it is crucial to automatically assign publications with their corresponding categories. Multi-label document classification (MLDC), associating one document instance with a set of most relevant la- bels, is attracting more and more research attention. To tackle the problem, early work focused on learning semantic representations of the input text using some text encoders. For example, Liu et al. ⇤Corresponding author. Venue NAACL Venue NAACL Reference ... Reference ... Reference Glove: Global Vectors ... Reference Glove: Global Vectors ... Reference Attention is All You Need Reference Attention is All You Need Author Jacob Devlin Author Jacob Devlin Author M-W Chang Author M-W Chang Author ... Author ... Document BERT: Pre- training of ... Document BERT: Pre- training of ... (a) Metadata Graph. Natural language processing Topic model Natural language Sequence labeling 0.66 0.62 0.62 0.54 0.33 Machine learning 0.43 0.40 (b) Label Graph. Figure 1: An example of a document with metadata and the label relationship in the MAG-CS dataset. In sub- figure (b), the arrow in black represents the hierarchy relationship between labels and the arrow in blue rep- resents the statistical dependency relationship between labels. (2017) proposed the XML-CNN model using a con- volutional neural network and You et al. (2018) pro- posed the AttentionXML model using a recurrent neural network as text semantic encoder. Recently, Chang et al. (2020) proposed …

Content truncated to 500 words