Task-oriented Domain-specific Meta-Embedding for Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3508–3513, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3508 Task-oriented Domain-specific Meta-Embedding for Text Classification Xin Wu1,2, Yi Cai1,2∗, Qing Li3, Tao Wang4, Kai Yang5 1School of Software Engineering, South China University of Technology, Guangzhou, China 2Key Laboratory of Big Data and Intelligent Robot (South China University of Technology), Ministry of Education 3Department of Computing, The Hong Kong Polytechnic University , Hong Kong, China 4Department of Biostatistics & Health Informatics, King’s College, London, UK 5City University of Hong Kong, Hong Kong, China ycai@scut.edu.cn Abstract Meta-embedding learning, which combines complementary information in different word embeddings, have shown superior perfor- mances across different Natural Language Processing tasks. However, domain-specific knowledge is still ignored by existing meta- embedding methods, which results in unstable performances across specific domains. More- over, the importance of general and domain word embeddings is related to downstream tasks, how to regularize meta-embedding to adapt downstream tasks is an unsolved prob- lem. In this paper, we propose a method to incorporate both domain-specific and task- oriented information into meta-embeddings. We conducted extensive experiments on four text classification datasets and the results show the effectiveness of our proposed method. 1 Introduction Building semantic representations (Zhao et al., 2017; Li et al., 2019; Neill and Bollegala, 2020) of words is a vital procedure in various Natural Language Processing (NLP) tasks. Over recent years, many pre-trained word embeddings have emerged, such as pre-trained Word2Vec (Mikolov et al., 2013) and pre-trained Glove (Pennington et al., 2014). Despite their usefulness, some pre- vious works find that the performance of different pre-trained word embeddings has significant vari- ation for different tasks (Chen et al., 2013; Hill et al., 2014). To obtain a stable and better per- formance, Yin and Sch¨utze (2015) proposed the meta-embedding learning task that aims to obtain a robust and superior word embedding (i.e., meta- embedding) by combining the different pre-trained word embeddings. Most previous meta-embedding methods neglect the importance of domain-specific information and ∗Corresponding author use the same embedding for each word in all domain-specific datasets (Bollegala and Bao, 2018; Coates and Bollegala, 2018; Bollegala et al., 2017). It is beneficial to incorporate domain-specific in- formation into general word embeddings and pro- vide different word representations for different domains, which has been shown to improve the performance in some other tasks (Bollegala et al., 2015; Xu et al., 2018). This leads us to explore how to combine gen- eral and domain-specific information in meta- embedding learning. Intuitively, the importance of the general and domain embeddings depends on a specific domain. For example, in the com- puter domain, for the domain-specific words (e.g., “mouse”), we should preserve their domain infor- mation but discard their general information. On the other hand, some general words (e.g., “we”, “people”) may not be able to get a high-quality do- main embedding due to the insufficient domain data, in this situation, their general word embed- dings are preferable. However, most previous meta- embedding methods …

Content truncated to 500 words