Evaluating Hierarchical Document Categorisation

Evaluating Hierarchical Document Categorisation Qian Sun♣ Aili Shen♣ Hiyori Yoshikawa♢ Chunpeng Ma♢ Daniel Beck♣ Tomoya Iwakura♢ Timothy Baldwin♣ ♣ The University of Melbourne ♢ Fujitsu Limited qiasun@student.unimelb.edu.au, {aili.shen, d.beck, tbaldwin}@unimelb.edu.au {y.hiyori, ma.chunpeng, iwakura.tomoya}@fujitsu.com Abstract Hierarchical document categorisation is a spe- cial case of multi-label document categorisa- tion, where there is a taxonomic hierarchy among the labels. While various approaches have been proposed for hierarchical document categorisation, there is no standard benchmark dataset, resulting in different methods being evaluated independently and there being no empirical consensus on what methods perform best. In this work, we examine different combi- nations of neural text encoders and hierarchical methods in an end-to-end framework, and eval- uate over three datasets. We find that the perfor- mance of hierarchical document categorisation is determined not only by how the hierarchical information is modelled, but also the structure of the label hierarchy and class distribution. 1 Introduction Document categorisation is a core task in infor- mation retrieval and natural language processing, whereby documents are categorised relative to a pre-defined set of labels. While the majority of research on document categorisation assumes a flat label structure, in practice in large-scale document categorisation tasks, there is often hierarchical la- bel structure, in the form of either a tree or directed acyclic graph (Zhou et al., 2020; Azarbonyad et al., 2021), where “child” labels inherit the properties of their parents. The goal of hierarchical document categorisation is to classify documents into a set of labels, where there is a hierarchical relationship among the labels. Hierarchical document categorisation methods explicitly capture the label structure during train- ing. There has been a resurgence of interest in doc- ument categorisation in recent years, in part driven by breakthroughs in representation learning and pre-trained language models (Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018; Kim, 2014; Wang et al., 2017; Devlin et al., 2019), which generate more expressive, general-purpose repre- sentations, thereby leading to performance gains across a range of NLP tasks. Despite this, there has been relatively little recent work specifically on hierarchical document categorisation. What recent work does has varied wildly in the choice of text encoder and dataset, with no systematic, controlled cross-dataset evaluation to be able to make solid conclusions as to whether the reported performance gains are attributable to the proposed hierarchical document categorisation method or just the text encoders used. Our work focuses on examining the capacity of existing methods dealing with la- bels with a hierarchical structure, which is different from the work of Yang et al. (2016), which focuses on modelling documents in a hierarchical way to perform classic document classification task. In this work, we carry out systematic evaluation of a range of contemporary hierarchical document categorisation approaches, using a range of neural text encoders, based on three document collections with hierarchical label sets. 2 Related Work Hierarchical document categorisation methods can be grouped into: flat approaches, local approaches, global approaches, and hybrid methods, based on how they utilise the label hierarchy. …

Content truncated to 500 words