Task-oriented Word Embedding for Text Classification

Proceedings of the 27th International Conference on Computational Linguistics, pages 2023–2032 Santa Fe, New Mexico, USA, August 20-26, 2018. 2023 Task-oriented Word Embedding for Text Classification Qian Liu1,2,3, Heyan Huang1,2∗, Yang Gao1,2, Xiaochi Wei1,2, Yuxin Tian1,2, Luyang Liu1,2 1. Department of Computer Science, Beijing Institute of Technology, China 2. Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, China 3. Centre for Artificial Intelligence, University of Technology Sydney, Australia Abstract Distributed word representation plays a pivotal role in various natural language processing tasks. In spite of its success, most existing methods only consider contextual information, which is suboptimal when used in various tasks due to a lack of task-specific features. The rational word embeddings should have the ability to capture both the semantic features and task-specific fea- tures of words. In this paper, we propose a task-oriented word embedding method and apply it to the text classification task. With the function-aware component, our method regularizes the distribution of words to enable the embedding space to have a clear classification boundary. We evaluate our method using five text classification datasets. The experiment results show that our method significantly outperforms the state-of-the-art methods. 1 Introduction Learning word representation is a fundamental step in various natural language processing tasks. Tremendous advances have been made by distributed representations (also known as word embeddings) which learn a transformation of each word from raw text data to a dense, lower-dimensional vector space. Most existing methods leverage contextual information from the corpus (Mikolov et al., 2013; Penning- ton et al., 2014) and other complementary information, such as subword information (Cao and Lu, 2017), implicitly syntactic dependencies (Shen et al., 2018a; Shen et al., 2018b), and semantic relations (Bolle- gala et al., 2016; Liu et al., 2018). In traditional evaluations such as word similarity and word analogy, the aforementioned context-aware word embeddings work well since semantic information plays a vital role in these tasks, and this infor- mation is naturally addressed by word contexts. However, in real-world applications, such as text classi- fication and information retrieval, word contexts alone are insufficient to achieve success in the absence of task-specific features. Figure 1 illustrates this problem with the classification task as an example. Sev- eral sentences from different categories are given at the far left of the figure where the words in bold are salient words for the category distinction. We also illustrated expected word distribution of these salient words in the embedding space. To obtain a good classification performance, the expected word distribu- tion should have a clear classification boundary: words within the same category are close to each other and far away from words in other categories as illustrated in Figure 1. However, the actual distribution obtained from Word2Vec at the far right of Figure 1 is normally not satisfactory because Word2Vec only focuses on context similarity. For example, although learning and educational are with similar context as recognized by Word2Vec, they are salient words to distinguish categories of AI and Sociology, …

Content truncated to 500 words