Evaluation of Sentence Representations in Polish

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1674–1680 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1674 Evaluation of Sentence Representations in Polish Sławomir Dadas, Michał Perełkiewicz, Rafał Po´swiata National Information Processing Institute, Warsaw, Poland {sdadas, mperelkiewicz, rposwiata}@opi.org.pl Abstract Methods for learning sentence representations have been actively developed in recent years. However, the lack of pre-trained models and datasets annotated at the sentence level has been a problem for low-resource languages such as Polish which led to less interest in applying these methods to language-specific tasks. In this study, we introduce two new Polish datasets for evaluating sentence embeddings and provide a comprehensive evaluation of eight sentence representation methods including Polish and multilingual models. We consider classic word embedding models, recently developed contextual embeddings and multilingual sentence encoders, showing strengths and weaknesses of specific approaches. We also examine different methods of aggregating word vectors into a single sentence vector. Keywords: sentence embedding, word embedding, polish language 1. Introduction In recent years, advances in representation learning for text have led to significant breakthroughs in the field of natural language processing (NLP). Word embeddings have been used with particular success in many linguistic tasks such as text classification, sequence tagging or machine transla- tion. Despite the wide adoption of word representations, especially in combination with deep learning architectures, some tasks require encoding larger chunks of text: sen- tences, paragraphs or documents. Information retrieval and plagiarism detection are common examples of problems that deal with longer texts and demand efficient methods of comparing or searching in large collections of documents. For this reason, there is an interest in the development of sentence and document embeddings. 1.1. Related work A straightforward and commonly used method for rep- resenting sentences is to aggregate the representations of words making up a sentence into a single vector. In this approach, existing word embedding methods can be used as a source of word vectors. Originally, the most popular choices for word embeddings were Word2Vec (Mikolov et al., 2013a; Mikolov et al., 2013b), GloVe (Pennington et al., 2014) or FastText (Bojanowski et al., 2017). More recently, it has been shown that neural language models trained on large corpora can be used to generate contextualized word representations that outperform static word embeddings in multiple linguistic tasks. Notable examples of such mod- els include ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), Flair (Akbik et al., 2018), ERNIE (Zhang et al., 2019), XLNet (Yang et al., 2019b). The sequence of word vectors is usually transformed into a single sentence vec- tor using Bag-of-Words (BoW) approach by computing an arithmetic or weighted mean but more complex approaches have been explored as well. Arora et al. (2017) proposed a method known as Smooth Inverse Frequency (SIF) which consisted of weighting word vectors and removing the pro- jection of first principal component from the resulting ma- trix of sentences. Shen et al. (2018) performed an anal- ysis of different aggregation techniques …

Content truncated to 500 words