AnlamVer: Semantic Model Evaluation Dataset for Turkish - Word Similarity and Relatedness

Proceedings of the 27th International Conference on Computational Linguistics, pages 3819–3836 Santa Fe, New Mexico, USA, August 20-26, 2018. 3819 AnlamVer: Semantic Model Evaluation Dataset for Turkish - Word Similarity and Relatedness Gökhan Ercan Department of Computer Engineering I¸sık University, ˙Istanbul, Turkey gokhan.ercan@isik.edu.tr Olcay Taner Yıldız Department of Computer Engineering I¸sık University, ˙Istanbul, Turkey olcaytaner@isikun.edu.tr Abstract In this paper, we present AnlamVer, which is a semantic model evaluation dataset for Turkish designed to evaluate word similarity and word relatedness tasks while discriminating those two relations from each other. Our dataset consists of 500 word-pairs annotated by 12 human sub- jects, and each pair has two distinct scores for similarity and relatedness. Word-pairs are selected to enable the evaluation of distributional semantic models by multiple attributes of words and word-pair relations such as frequency, morphology, concreteness and relation types (e.g., syn- onymy, antonymy). Our aim is to provide insights to semantic model researchers by evaluating models in multiple attributes. We balance dataset word-pairs by their frequencies to evaluate the robustness of semantic models concerning out-of-vocabulary and rare words problems, which are caused by the rich derivational and inflectional morphology of the Turkish language. Title and Abstract in Turkish AnlamVer: Anlambilimsel Model Ölçümleme Veri Kümesi - Kelime Benzerli˘gi ve ˙Ili¸skiselli˘gi Bu makalede, Türkçe için kelime benzerlik ve ili¸skisellik görevlerini ayrı ayrı ölçümleyebilmek için tasarlanmı¸s anlambilimsel bir model veri kümesi olan AnlamVer’i sunuyoruz. Veri kümemiz, 12 ki¸si tarafından her kelime çifti için ili¸skisellik ve benzerlik puanları ayrı ayrı puanlanmı¸s toplam 500 kelime çiftinden olu¸smaktadır. Kelime çiftleri, da˘gılımsal anlambilimsel modelleri kelimelerin ve kelime çiftlerinin sıklık, morfoloji, somutluk ve ili¸ski tipleri (e¸s anlamlılık, kar¸sıt anlamlılık vb.) gibi birden fazla niteli˘gine göre ölçümleyebilmek için seçilmi¸stir. Amacımız, anlambilimsel model ara¸stırmacılarının modellerini birden fazla niteli˘ge göre ölçümleyerek içgörüler kazanmasını sa˘glamaktır. Veri kümesindeki kelime çiftleri, Türkçe’nin zengin türetim- sel ve çekimsel yapısı kaynaklı sözlük-dı¸sı ve seyrek-kelime problemlerine kar¸sı anlambilimsel modellerin sa˘glamlı˘gını ölçümleyebilmek amacıyla sıklık de˘gerlerine göre dengelenmi¸stir. 1 Introduction Unsupervised semantic modeling has recently gained a lot of attention in the NLP community due to the notion of high reusability of pre-trained models across a variety of higher level NLP tasks such as machine translation, word sense disambiguation and named entity recognition. Increasing computabil- ity of unsupervised distributional semantic modeling (DSM) methods enable researchers to increase the performance of NLP tasks by leveraging extracted semantic information from a high volume of un- structured texts at low costs. However, there are very few available methods and resources to evaluate semantic models intrinsically regardless of the higher level tasks’ dynamics. Presenting word similarity and word relatedness (i.e., association) dataset AnlamVer, we aim at providing the semantic modeling field for Turkish with an intrinsic evaluation resource targeting morphology driven issues caused by the rich agglutinative nature of the language. We are not aware of the existence of such word similarity or This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 3820 relatedness evaluation resources constructed for Turkish. In this paper, we describe design considerations and …

Content truncated to 500 words