COSTRA 1.0: A Dataset of Complex Sentence Transformations

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3535–3541 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3535 COSTRA 1.0: A Dataset of Complex Sentence Transformations Petra Baranˇc´ıkov´a, Ondˇrej Bojar Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {barancikova,bojar}@ufal.mff.cuni.cz Abstract We present COSTRA 1.0, a dataset of complex sentence transformations. The dataset is intended for the study of sentence-level embeddings beyond simple word alternations or standard paraphrasing. This first version of the dataset is limited to sentences in Czech but the construction method is universal and we plan to use it also for other languages. The dataset consists of 4,262 unique sentences with an average length of 10 words, illustrating 15 types of modifications, such as simplification, generalization, or formal and informal language variation. The hope is that with this dataset, we should be able to test semantic properties of sentence embeddings and perhaps even to find some topologically interesting “skeleton” in the sentence embedding space. A preliminary analysis using LASER, multi-purpose multi-lingual sentence embeddings suggests that the LASER space does not exhibit the desired properties. Keywords: sentence embeddings, sentence transformations, paraphrasing, semantic relations 1. Introduction Vector representations are essential in the majority of nat- ural language processing tasks. The popularity of word embeddings started with the introduction of word2vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014) and their properties have been analyzed at length from var- ious aspects. Studies of word embeddings range from word similarity (Hill et al., 2014; Faruqui and Dyer, 2014), over the ability to capture derivational relations (Musil et al., 2019), linear superposition of multiple senses (Arora et al., 2016), the ability to predict semantic hierarchies (Fu et al., 2014) or POS tags (Musil, 2019) up to data efficiency (Jastrzkebski et al., 2017). Several studies (Mikolov et al., 2013c; Mikolov et al., 2013b; Levy and Goldberg, 2014; Vylomova et al., 2015) show that word vector representations are capable of capturing meaningful syntactic and semantic regular- ities. These include, for example, male/female relation demonstrated by the pairs “man:woman”, “king:queen” and the country/capital relation (“Russia:Moscow”, “Japan:Tokyo”). These regularities correspond to simple arithmetic operations in the vector space. Sentence embeddings are becoming equally ubiquitous in NLP, with novel representations appearing almost ev- ery other week. With an overwhelming number of meth- ods to compute sentence vector representations, the study of their general properties becomes difficult. Furthermore, it is not entirely clear in which way the embeddings should be evaluated. In an attempt to bring together more traditional representa- tions of sentence meanings and the emerging vector repre- sentations, Bojar et al. (2019) introduce several aspects or desirable properties of sentence embeddings. One of them, “relatability”, highlights the correspondence of meaningful differences between sentences on the one hand and geomet- rical relations between their respective embeddings in the highly dimensional continuous vector space on the other hand. If we found such correspondence, we could apply geometrical operations in the space to …

Content truncated to 500 words