Text Categorization by Learning Predominant Sense of Words as Auxiliary Task

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1109–1119 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1109 Text Categorization by Learning Predominant Sense of Words as Auxiliary Task Kazuya Shimura1, Jiyi Li2,3 and Fumiyo Fukumoto2 Graduate School of Engineering, University of Yamanashi1 Interdisciplinary Graduate School, University of Yamanashi2 4-3-11, Takeda, Kofu, 400-8511 Japan RIKEN AIP3, Tokyo, 103-0027 Japan {g17tk008,jyli,fukumoto}@yamanashi.ac.jp Abstract Distributions of the senses of words are often highly skewed and give a strong influence of the domain in a document. This paper fol- lows the assumption and presents a method for text categorization by leveraging the predomi- nant sense of words depending on the domain, i.e., domain-specific senses. The key idea is that the features learned from predominant senses are possible to discriminate the domain of the document and thus improve the overall performance of text categorization. We pro- pose a multi-task learning framework based on the neural network model, transformer, which trains a model to simultaneously categorize documents and predicts a predominant sense for each word. The experimental results us- ing four benchmark datasets including RCV1 show that our method is comparable to the state-of-the-art categorization approach, espe- cially our model works well for categorization of multi-label documents. 1 Introduction Text categorization has been intensively studied since neural network methods have attracted much attention. Most of the previous work on text categorization relies on the use of representation learning where the words are mapped to an im- plicit semantic space (Wang et al., 2015; Liu et al., 2017a). The Word2Vec is a typical model related to this representation (Mikolov et al., 2013). learns a vector representation for each word andIt captures semantic information between words. Pre-training by using the model shows that it im- proves overall performance in many NLP tasks in- cluding text categorization. However, the draw- back in the implicit representation is that it often does not work well on polysemous words. The sense of a word depends on the domain in which it is used. The same word can be used dif- ferently in different domains. Distributions of the senses of words are often highly skewed and a pre- dominant sense of a word depends on the domain of a document (McCarthy et al., 2007; Jin et al., 2009). Suppose the noun word, “court”. The pre- dominant sense of a word “court” would be dif- ferent in the documents from the “judge/law” and “sports” domains as the sense of the former would be “an assembly (including one or more judges) to conduct judicial business” and the latter is “a spe- cially marked horizontal area within which a game is played” described in the WordNet 3.1. This in- dicates that the meaning becomes a strong clue to assign a domain to the document. However, in the implicit semantic space created by using the neu- ral language model such as the Word2Vec, a word is represented as one vector even if it has several senses. It …

Content truncated to 500 words