Transformation of Dense and Sparse Text Representations

Proceedings of the 28th International Conference on Computational Linguistics, pages 3257–3267 Barcelona, Spain (Online), December 8-13, 2020 3257 Transformation of Dense and Sparse Text Representations Wenpeng Hu1,∗, Mengyu Wang2,∗, Bing Liu2,†, Feng Ji3, Jinwen Ma1, Dongyan Zhao2 1 Department of Information Science, Peking University 2 Wangxuan Institute of Computer Technology, Peking University 3 Alibaba Group {wenpeng.hu, wangmengyu, dcsliub, jwma, zhaody}@pku.edu.cn zhongxiu.jf@alibaba-inc.com Abstract Sparsity is regarded as a desirable property of representations, especially in terms of explanation. However, its usage has been limited due to the gap with dense representations. Most research progresses in NLP in recent years are based on dense representations. Thus the desirable property of sparsity cannot be leveraged. Inspired by Fourier Transformation, in this paper, we propose a novel Semantic Transformation method to bridge the dense and sparse spaces, which can facil- itate the NLP research to shift from dense spaces to sparse spaces or to jointly use both spaces. Experiments using classification tasks and natural language inference task show that the proposed Semantic Transformation is effective. 1 Introduction A sparse vector is a vector that has a large number of zeros or near zeros. Many studies have shown that sparsity is a desirable property of representations, especially for explanation (Fyshe et al., 2014; Faruqui and Dyer, 2015). In this sense, sparse representation may hold the key to solving the explainability problem of deep neural networks. Apart from the interpretability property, sparse representations can also improve the usability of word vectors as features. The embeddings with good sparsity, interpretability or special meanings can also benefit downstream tasks (Guo et al., 2014; Chang et al., 2018). Several tasks have benefited from sparse representations, e.g., part-of-speech tagging (Ganchev et al., 2009), dependency parsing (Martins et al., 2011), and supervised classification (Yogatama and Smith, 2014). However, much of the research advances so far for NLP tasks are based on dense representations, e.g., text classification (Kim, 2014; Tang et al., 2015; Wu et al., 2017; Wang et al., 2018), natural language inference (Liu et al., 2019; Kim et al., 2019), machine translation (Cheng, 2019; He et al., 2016) and generation (Serban et al., 2017; Zhang et al., 2019; Du and Cardie, 2017). The study of sparse representations is still limited. There are two key limitations in the study of sparse representations. First, little work has been done to well connect dense and sparse spaces. The two types of representation are rather independent and cannot help each other to achieve synergy. Second, limited work has been done to generate representations of sentences or phrases in the sparse space using sparse word embeddings. Inspired by Fourier Transformation, this paper proposes a novel method called Semantic Transforma- tion (ST) to address the problems. With the help of ST, dense and sparse spaces can connect with each other and will not be isolated. The proposed transformation consists of two key components, namely, Semantic Forward Transformation (SFT) and Semantic Backward Transformation (SBT) (see Section 2). SFT is designed to transform a dense representation to a sparse …

Content truncated to 500 words