Word Embedding Evaluation for Sinhala

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1874–1881 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1874 Word Embedding Evaluation for Sinhala Dimuthu Lakmal, Surangika Ranathunga, Saman Peramuna, Indu Herath Department of Computer Science and Engineering, University of Moratuwa Katubedda 10400, Sri Lanka (kjtdimuthu.13, surangika)@cse.mrt.ac.lk, (peramunas, induh)@uom.lk Abstract This paper presents the first ever comprehensive evaluation of different types of word embeddings for Sinhala language. Three standard word embedding models, namely, Word2Vec (both Skipgram and CBOW), FastText, and Glove are evaluated under two types of evaluation methods: intrinsic evaluation and extrinsic evaluation. Word analogy and word relatedness evaluations were performed in terms of intrinsic evaluation, while sentiment analysis and part-of-speech (POS) tagging were conducted as the extrinsic evaluation tasks. Benchmark datasets used for intrinsic evaluations were carefully crafted considering specific linguistic features of Sinhala. In general, FastText word embeddings with 300 dimensions reported the finest accuracies across all the evaluation tasks, while Glove reported the lowest results. Keywords: Word Embedding, Sinhala, Evaluation Methodologies 1. Introduction Distributed representation of words, commonly known as word embeddings, can be considered as one of the preliminary building blocks for most of the down- stream Natural Language Processing (NLP) tasks in this era. Word embeddings enable the exploration of fine-grained semantic and syntactic relationships among words by representing each feature as a vector in a low dimensional space (Goldberg, 2017; Elraz- zaz et al., 2017). Researchers have been using various distributed representations of words in natural lan- guage processing paradigms for decades now. With the rapid growth of neural network based deep learn- ing techniques in recent years, novel neural word em- bedding approaches have been introduced to overcome the issues of traditional distributional word representa- tions such as Latent Semantic Analysis (LSA). These memory efficient, dense neural word embeddings sur- pass the performance of traditional sparse word vec- tor representations (Bengio et al., 2003; Mikolov et al., 2013a). Moreover, some researchers obtained word embeddings by combining statistical features used in traditional word embeddings with the features of re- cent neural word embedding models to improve the performance (Pennington et al., 2014). NLP community relies on two types of evaluation pro- cedures for word embeddings: intrinsic and extrinsic evaluation. Intrinsic evaluations directly test for syn- tactic or semantic relationships between words (Schn- abel et al., 2015). Intrinsic evaluations are conducted using the datasets crafted based on human assessments on word relations (Bakarov, 2018). Each entry of these datasets consists of related words and a target word or quantified relatedness among words. On the other hand, extrinsic evaluation tasks measure the performance of word embeddings by using them as inputs to downstream NLP tasks such as senti- ment classification, Part-of-Speech (POS) tagging, and Named Entity Recognition (NER) (Pennington et al., 2014; Turian et al., 2010). Even though there is a growing interest in evaluat- ing different word embeddings based on various lin- guistic and downstream NLP tasks, only a handful of studies have been conducted …

Content truncated to 500 words