Sinhala Sentence Embedding: A Two-Tiered Structure for Low-Resource Languages

Sinhala Sentence Embedding: A Two-Tiered Structure for Low-Resource Languages Gihan Weeraprameshwara*,1, Vihanga Jayawickrama*, Nisansa de Silva*, and Yudhanjaya Wijeratne** *Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka **LIRNEasia, Sri Lanka 1gihanravindu.17@cse.mrt.ac.lk Abstract In the process of numerically modeling natural languages, developing language embeddings is a vital step. However, it is challenging to develop functional embeddings for resource- poor languages such as Sinhala, for which sufficiently large corpora, effective language parsers, and any other required resources are difficult to find. In such conditions, the ex- ploitation of existing models to come up with an efficacious embedding methodology to nu- merically represent text could be quite fruitful. This paper explores the effectivity of several one-tiered and two-tiered embedding architec- tures in representing Sinhala text in the senti- ment analysis domain. With our findings, the two-tiered embedding architecture where the lower-tier consists of a word embedding and the upper-tier consists of a sentence embedding has been proven to perform better than one-tier word embeddings, by achieving a maximum F1 score of 88.04% in contrast to the 83.76% achieved by word embedding models. Further- more, embeddings in the hyperbolic space are also developed and compared with Euclidean embeddings in terms of performance. A senti- ment data set consisting of Facebook posts and associated reactions have been used for this re- search. To effectively compare the performance of different embedding systems, the same deep neural network structure has been trained on sentiment data with each of the embedding sys- tems used to encode the text associated. 1 Introduction An effective numerical representation of the textual content is crucial for natural language processing models, in order to understand the underlying rela- tional patterns among words and discover patterns in natural languages. For resource-rich languages like English, numerous pre-trained models as well as the required materials to develop an embedding system are readily available. On the contrary, for resource-poor languages such as Sinhala, neither of those options could be easily found (de Silva, 2019). Even the data sets that are available for training of- ten fail to meet adequate standards (Caswell et al., 2021). Thus, discovering a convenient method- ology to develop embeddings for text would be a great step forward in the NLP domain for the Sinhala language. Sinhala, also known as Sinhalese, is an Indo- Aryan language that is used within Sri Lanka (Kan- duboda, 2011). The primary user base of this lan- guage is the Sinhalese ethnic group of the country. In total, 17 million people use Sinhala as their first language while 2 million people use it as a second language (de Silva, 2019). Furthermore, Sinhala is structurally different from English, which uses a subject-verb-object structure as opposed to the subject-object-verb structure used by Sinhala as shown in the figure 1 thus most of the pre-trained embedding models for English may not be effective with Sinhala. Figure 1: SVO grammar structure of English and SOV grammar structure of Sinhala This study therefore is focused on discovering an effective embedding system …

Content truncated to 500 words