Leveraging Linguistic Resources for Improving Neural Text Classification

Leveraging linguistic resources for improving neural text classification Ming Liu FIT, Monash University ming.m.liu@monash.edu Gholamreza Haffari FIT, Monash University gholamreza.haffari@monash.edu Wray Buntine FIT, Monash University wray.buntine@monash.edu Michelle R. Ananda-Rajah Alfred Health and Monash University michelle.ananda-rajah@monash.edu Abstract This paper presents a deep linguistic atten- tional framework which incorporates word level concept information into neural clas- sification models. While learning neural classification models often requires a large amount of labelled data, linguistic concept information can be obtained from exter- nal knowledge, such as pre-trained word embeddings, WordNet for common text and MetaMap for biomedical text. We explore two different ways of incorporat- ing word level concept annotations, and show that leveraging concept annotations can boost the model performance and re- duce the need for large amounts of labelled data. Experiments on various data sets validate the effectiveness of the proposed method. 1 Introduction Text classification is an important task in natural language processing, such as sentiment analysis, information retrieval, web page ranking and doc- ument classification (Pang et al., 2008). Recently, deep neural models have been widely used in this area due to their abstract framework and good per- formance. While these models are being used fre- quently, they require a large amount of labelled data and training time. The core idea of text neural classification mod- els is that text signals are fed into composition and activation functions via deep neural networks, and then a softmax classifier generates the final label as a probability distribution. Unlike standard n- gram models, word representation (Mikolov et al., 2013) is distributed and manual features are not usually necessary in deep neural models. Though promising, most current text neural classification models still lack the ability of mod- eling linguistic information of the language, es- pecially in domains where annotations are time- consuming and expensive such as biomedical text. In this work, we use some prior knowledge from pre-trained word embeddings or knowledge bases, and explore different ways of incorporating this prior knowledge into existing deep neural classi- fication models. Our model is an integration of a simple neural bag of words model, which works in 2 steps: 1. create mappings from a sequence of word to- kens into concept tokens (based on the given pre-trained word embeddings or knowledge bases), 2. combine the embeddings of both word and concept tokens and pass the resulting embed- ding through a deep feed-forward classifica- tion model to make the final prediction. The motivation of our work is to incorporate ex- tra knowledge from pre-trained word embeddings or knowledge bases such as WordNet for common text, MetaMap for biomedical text. Our main con- tributions are: (1) creating linguistically-related concepts of words from external knowledge bases; (2) incorporating the concept information either through what we call direct or gated mappings. We show that leveraging concept annotations can boost the model performance and reduce the need for large amounts of labelled data, and the concept information can be incorporated more effectively in a gated mapping manner. The remainder of this paper …

Content truncated to 500 words