Deep Belief Networks and Biomedical Text Categorisation

Deep Belief Networks and Biomedical Text Categorisation Antonio Jimeno Yepes♦♠, Andrew MacKinlay♦♠, Justin Bedo♦♠, Rahil Garnavi♦, Qiang Chen♦ ♦ IBM Research – Australia, 380 La Trobe Street, Melbourne, VIC, Australia ♠ Dept. of Computing and Information Systems, University of Melbourne, Australia {antonio.jimeno,admackin,justin.bedo,rahilgar,qiangchen}@au1.ibm.com Abstract We evaluate the use of Deep Belief Net- works as classifiers in a text categorisa- tion task (assigning category labels to doc- uments) in the biomedical domain. Our preliminary results indicate that compared to Support Vector Machines, Deep Belief Networks are superior when a large set of training examples is available, showing an F-score increase of up to 5%. In addition, the training times for DBNs can be pro- hibitive. DBNs show promise for certain types of biomedical text categorisation. 1 Introduction Text categorisation is the task of automatically as- signing pre-defined labels to text. In the biomedical domain, research in automatic text categorisation has mostly taken place in the context of indexing MEDLINE R⃝ citations with Medical Subject Head- ings (MeSH R⃝). MEDLINE is the largest collection of biomed- ical abstracts and contains over 23M citations with over 800k new citations every year, making it difficult to keep up-to-date with new discover- ies. To help cataloging and searching biomedical documents, the US National Library of Medicine (NLM) R⃝ has produced the MeSH controlled vo- cabulary with over 26k headings. At NLM, each MEDLINE citation is manually assigned a number of relevant medical subject headings enumerating the topics of the paper. Machine learning for text categorisation in this context is appealing due to the large number of citations available to train machine learning algorithms. In text categorisation, the most frequently used feature representation is bag-of-words, where text is converted into a feature vector in which each dimension corresponds to a word or phrase and stores either a binary value indicating its presence in the document or a numerical value indicating its frequency (Apte et al., 1994; Dumais et al., 1998; Sebastiani, 2002). This relatively simple approach has proven to be robust enough (Jimeno-Yepes et al., 2011) that it is difficult to improve on its perfor- mance with more sophisticated representations. In this work, we explore the use of Deep Belief Net- works (DBN) to automatically generate a new rep- resentation in biomedical text categorisation. Since DBNs have a richer internal representation than SVMs, we wished to evaluate whether this would lead to improved classification performance com- pared to bag-of-words. 2 Related work There are several approaches being used for text categorisation in the biomedical domain trying to reproduce the manual MeSH indexing. NLM has developed the Medical Text Indexer (MTI) (Aron- son et al., 2004; Mork et al., 2013), which is used to suggest MeSH headings for new citations to indexers. MTI combines MetaMap (Aronson and Lang, 2010) annotation and recommendations from similar citations recovered using the PubMed Re- lated Citations (Lin and Wilbur, 2007) tool that are post-processed to comply with NLM indexing rules. Results for the most frequent categories, as used in this work, indicate …

Content truncated to 500 words