Term Based Semantic Clusters for Very Short Text Classification

Proceedings of Recent Advances in Natural Language Processing, pages 878–887, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_102 878 Term Based Semantic Clusters for Very Short Text Classification Jasper Paalman Jheronimus Academy of Data Science j.v.paalman@tilburguniversity.edu Shantanu Mullick School of Industrial Engineering Eindhoven University of Technology s.mullick@tue.nl Kalliopi Zervanou School of Industrial Engineering Eindhoven University of Technology k.zervanou@tue.nl Yingqian Zhang School of Industrial Engineering Eindhoven University of Technology yqzhang@tue.nl Abstract Very short texts, such as tweets and in- voices, present challenges in classification. Although term occurrences are strong in- dicators of content, in very short texts, the sparsity of these texts makes it difficult to capture important semantic relationships. A solution calls for a method that not only considers term occurrence, but also han- dles sparseness well. In this work, we in- troduce such an approach, the Term Based Semantic Clusters (TBSeC) that employs terms to create distinctive semantic con- cept clusters. These clusters are ranked using a semantic similarity function which in turn defines a semantic feature space that can be used for text classification. Our method is evaluated in an invoice classifi- cation task. Compared to well-known con- tent representation methods the proposed method performs competitively. 1 Introduction Bag-of-words approaches (Harris, 1954) to text classification rely on measures of term occurrence, or co-occurrence, such as tf ·idf (Salton and Buck- ley, 1988), log-likelihood (Dunning, 1993), and mutual information (Church and Hanks, 1990). Such methods, despite their enduring popularity, are well known for their shortcomings in deal- ing with numerous natural language issues, such as morphological, semantic, and other types of variation and ambiguity (Augenstein et al., 2017). These issues become more critical in very short texts, such as microblogs, chat logs, reviews and invoices, because of the lack of data and context that could provide more reliable measures and a source for semantic disambiguation. Distributional semantic models (Baroni and Lenci, 2010) such as pre-trained word embeddings (Pennington et al., 2014; Grave et al., 2018) encode words as fixed sized real-valued vectors. Embed- dings may address the issues of data sparseness and lack of extensive context, by providing a semantic representation model that is not as sensitive to lit- eral word occurrence in the data, thus providing se- mantic information coverage of out-of-vocabulary words (Bojanowski et al., 2017). This is because embeddings may allow for any given word to be mapped to a real-valued vector (e.g. by using char- acter n-grams), even if it hasn’t been observed dur- ing training. Additionally, embeddings implicitly capture semantic, syntactic and lexical properties, thereby representing implicit relationships. In this way, embeddings provide a rich representation that is otherwise difficult to attain. In a short text sce- nario, despite early findings that a small corpus size may not be sufficient for representing the re- lationships between words and documents (Song et al., 2014), availability of pre-trained embeddings makes this issue less of a concern. Despite these advantages and growing popularity, embeddings trained on general language data usually do not perform well in (i) specialised domains (Liu …

Content truncated to 500 words