Categorical Metadata Representation for Customized Text Classification

Categorical Metadata Representation for Customized Text Classification Jihyeok Kim*1 Reinald Kim Amplayo*2 Kyungjae Lee1 Sua Sung1 Minji Seo1 Seung-won Hwang1 (* equal contribution) 1Yonsei University zizi1532@yonsei.ac.kr2University of Edinburgh reinald.kim@ed.ac.uk {lkj0509,dormouse,ggatalminji,seungwonh}@yonsei.ac.kr Abstract The performance of text classification has improved tremendously using intelligently engineered neural-based models, especially those injecting categorical metadata as additional information, e.g., using user/product informa- tion for sentiment classification. This infor- mation has been used to modify parts of the model (e.g., word embeddings, attention mech- anisms) such that results can be customized according to the metadata. We observe that current representation methods for categorical metadata, which are devised for human con- sumption, are not as effective as claimed in popular classification methods, outperformed even by simple concatenation of categorical features in the final layer of the sentence encoder. We conjecture that categorical fea- tures are harder to represent for machine use, as available context only indirectly describes the category, and even such context is often scarce (for tail category). To this end, we pro- pose using basis vectors to effectively incor- porate categorical metadata on various parts of a neural-based model. This additionally decreases the number of parameters dramatic- ally, especially when the number of categori- cal features is large. Extensive experiments on various data sets with different properties are performed and show that through our method, we can represent categorical metadata more effectively to customize parts of the model, including unexplored ones, and increase the performance of the model greatly. 1 Introduction Text classification is the backbone of most NLP tasks: review classification in sentiment analysis (Pang et al., 2002), paper classification in sci- entific data discovery (Sebastiani, 2002), and question classification in question answering (Li and Roth, 2002), to name a few. While prior meth- ods require intensive feature engineering, recent methods enjoy automatic extraction of features from text using neural-based models (Socher et al., 2011) by encoding texts into low-dimensional dense feature vectors. This paper discusses customized text clas- sification, generalized from personalized text classification (Baruzzo et al., 2009), where we customize classifiers based on possibly multiple different known categorical metadata information (e.g., user/product information for sentiment classification) instead of just the user information. As shown in Figure 1, in addition to the text, a customizable text classifier is given a list of categories specific to the text to predict its class. Existing works applied metadata information to improve the performance of a model, such as user and product (Tang et al., 2015) information in sentiment classification, and author (Rosen- Zvi et al., 2004) and publication (Joorabchi and Mahdi, 2011) information in paper classification. Towards our goal, we are inspired by the ad- vancement in neural-based models, incorporat- ing categorical information ‘‘as is’’ and injecting it on various parts of the model such as in the word embeddings (Tang et al., 2015), attention mechanism (Chen et al., 2016; Amplayo et al., 2018a) and memory networks (Dou, 2017). Indeed, these methods theoretically make use of combined features from both textual and categorical features, which make them …

Content truncated to 500 words