Polish evaluation dataset for compositional distributional semantics models

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 784–792 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1073 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 784–792 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1073 Polish evaluation dataset for compositional distributional semantics models Alina Wróblewska Katarzyna Krasnowska-Kiera´s Institute of Computer Science, Polish Academy of Sciences alina@ipipan.waw.pl kasia.krasnowska@gmail.com Abstract The paper presents a procedure of building an evaluation dataset1. for the validation of compositional distributional semantics models estimated for languages other than English. The procedure generally builds on steps designed to assemble the SICK corpus, which contains pairs of English sentences annotated for semantic related- ness and entailment, because we aim at building a comparable dataset. However, the implementation of particular building steps significantly differs from the orig- inal SICK design assumptions, which is caused by both lack of necessary extra- neous resources for an investigated lan- guage and the need for language-specific transformation rules. The designed proce- dure is verified on Polish, a fusional lan- guage with a relatively free word order, and contributes to building a Polish evalu- ation dataset. The resource consists of 10K sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for the evaluation of compositional distributional semantics models of Polish. 1 Introduction and related work 1.1 Distributional semantics The basic idea of distributional semantics, i.e. de- termining the meaning of a word based on its co-occurrence with other words, is derived from the empiricists – Harris (1954) and Firth (1957). John R. Firth drew attention to the context- dependent nature of meaning especially with his 1The dataset is obtainable at: http://zil.ipipan.waw.pl/Scwad/CDSCorpus famous maxim “You shall know a word by the company it keeps” (Firth, 1957, p. 11). Nowadays, distributional semantics models are estimated with various methods, e.g. word em- bedding techniques (Bengio et al., 2003, 2006; Mikolov et al., 2013). To ascertain the purport of a word, e.g. bath, you can use the context of other words that surround it. If we assume that the meaning of this word expressed by its lexical context is associated with a distributional vector, the distance between distributional vectors of two semantically similar words, e.g bath and shower, should be smaller than between vectors represent- ing semantically distinct words, e.g. bath and tree. 1.2 Compositional distributional semantics Based on empirical observations that distribu- tional vectors encode certain aspects of word meaning, it is expected that similar aspects of the meaning of phrases and sentences can also be represented with vectors obtained via composi- tion of distributional word vectors. The idea of se- mantic composition is not new. It is well known as the principle of compositionality:2 “The mean- ing of a compound expression is a function of the meaning of its parts and of the way they are syntactically combined.” (Janssen, 2012, p. 19). Modelling the meaning of textual units larger than …

Content truncated to 500 words