Evaluation of vector embedding models in clustering of text documents

Proceedings of Recent Advances in Natural Language Processing, pages 1304–1311, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_149 1304 Evaluation of vector embedding models in clustering of text documents Tomasz Walkowiak University of Science and Technology, Faculty of Electronics, Wybrzeze Wyspianskiego 27, Wroclaw 50-370, Poland tomasz.walkowiak@pwr.wroc.pl Mateusz Gniewkowski University of Science and Technology, Faculty of Computer Science and Management, Wybrzeze Wyspianskiego 27, Wroclaw 50-370, Poland mateusz.gniewkowski@pwr.wroc.pl Abstract The paper presents an evaluation of word embedding models in clustering of texts in the Polish language. Authors verified six different embedding models, starting from widely used word2vec, across fast- Text with character n-grams embedding, to deep learning-based ELMo and BERT. Moreover, four standardisation methods, three distance measures and four cluster- ing methods were evaluated. The analy- sis was performed on two corpora of texts in Polish classified into subjects. The Ad- justed Mutual Information (AMI) metric was used to verify the quality of cluster- ing results. The performed experiments show that Skipgram models with n-grams character embedding, built on KGR10 cor- pus and provided by Clarin-PL, outper- forms other publicly available models for Polish. Moreover, presented results sug- gest that Yeo–Johnson transformation for document vectors standardisation and Ag- glomerative Clustering with a cosine dis- tance should be used for grouping of text documents. 1 Introduction A number of digital repositories of texts enlarge each year. The variety of tools for natural lan- guage processing and the quality of their perfor- mance are growing steadily. That opens possibili- ties for automatic categorisation of text documents in terms of the subject areas in any digital collec- tion of documents. It is also an important problem for researchers from different areas of the human- ities and social science (Eder et al., 2017). Commonly used methods rely on representing documents with feature vectors and using clus- tering algorithm (Hastie et al., 2009) to assign documents to some groups. The classical feature vectors are based on the bag-of-words technique (Harris, 1954). Components of these vectors rep- resent frequencies (weighted) of occurrences of words/terms in individual documents. The con- temporary state-of-the-art technique is word2vec (Le and Mikolov, 2014), where individual words are represented by high-dimensional feature vec- tors trained on large text corpora. This technique is constantly being improved. This is demonstrated by the most recent propositions of algorithms like ELMo (Peters et al., 2018) or BERT (Devlin et al., 2018). Choosing the most useful clustering algo- rithm is not a trivial task since there is a large num- ber of them. Just to mention the most popular ones like K-means (K.Jain, 2009), Agglomerative Hier- archical Clustering (Day and Edelsbrunner, 1984) and Spectral Clustering (Ng et al., 2002). More- over, the results of clustering are strongly depen- dent on the chosen distance measure and the used method of an input data standardisation. The en- tire workflow, described above, expresses many factors which can influence results of the text ex- ploration. It is difficult to control them and thus, might lead to unpredictable outcomes of the ex- periment. It becomes challenging …

Content truncated to 500 words