SimLex-999 for Polish

SimLex-999 for Polish Małgorzata Marciniak1, Agnieszka Mykowiecka1,2, Piotr Rychlik1 Institute of Computer Science, Polish Academy of Sciences1; Polish-Japanese Academy of Information Technology2 Jana Kazimierza 5, Warsaw, Poland; Koszykowa 86, Warsaw, Poland {mm, agn, rychlik}@ipipan.waw.pl Abstract The paper addresses the Polish version of SimLex-999 which we extended to contain not only measurement of similarity but also relatedness. The data was translated by three independent linguists; discrepancies in translation were resolved by a fourth person. The agreement rates between the translators were counted and an analysis of problems was performed. Then, pairs of words were rated by other annotators on a scale of 0–10 for similarity and relatedness of words. Finally, we compared the human annotations with the distributional semantics models of Polish based on lemmas and forms. We compared our work with the results reported for other languages. Keywords: similarity, synonymy, Polish, distributional models evaluation 1. Introduction Distributional semantics methods are commonly used in various linguistic tasks. So creating resources to evaluate them is extremely important. The best known of these, (Gabrilovich, 2017) (Finkelstein et al., 2002), consists of 353 word pairs together with their similarity scores. Human annotators evaluated the similarity of word pairs under- stood as synonymy, relatedness and association of words. Another well-known resource developed for the same pur- pose is SimLex-999 (Hill, 2017) (Hill et al., 2015). It was created to test similarity, but here it was understood as syn- onymy and quasi-synonymy, which seems to be a hyper- nym/hyponym relation or co-hyponymy. It consists of 999 word pairs rated for this interpretation of similarity. The authors clearly excluded relatedness and association from the similarity relation. Moreover, the resource provides in- formation on part-of-speech, abstract vs. concrete concepts, and independent measures of relatedness of pairs of words for English. Both the above resources were translated into German, Italian and Russian (Leviant and Reichart, 2015) and are available as Multilingual WS353 and Multilingual SimLex999 (Leviant and Reichart, 2017), respectively. 1 Many authors (e.g. (Faruqui et al., 2016), (Chiu et al., 2016)) point out the disadvantages of evaluation for distri- butional semantics methods based on isolated tests which are not connected to a whole processing system. For lan- guages with less developed linguistic infrastructure, such as Polish, it is important to provide resources for intrinsic eval- uation too. Therefore, we decided to translate SimLex-999. As it would be interesting to compare results for synonymy and relatedness on the same resource, we rated both types of similarity relation. 2. Translation The translation of SimLex-999 into Polish was done by three linguists, native speakers of Polish with good knowl- edge of English, according to the instruction published to- gether with Multilingual SimLex999. The final translation 1The list of English words was copied from the original SimLex-999 set but new similarity values were assigned. was agreed by a fourth person with a computational linguis- tics background. She only resolved cases where differences in the three translations occurred, for 585 pairs in fact. As the Multilingual SimLex999 resource includes translation into Russian, a …

Content truncated to 500 words