B2SG: a TOEFL-like Task for Portuguese

B2SG: a TOEFL-like Task for Portuguese Rodrigo Wilkens, Leonardo Zilio, Eduardo Ferreira, Aline Villavicencio Institute of Informatics, Federal University of Rio Grande do Sul (Brazil) rodrigo.wilkens, lzilio@inf.ufrgs.br, eduardo.ferreira, avillavicencio@inf.ufrgs.br Abstract Resources such as WordNet are useful for NLP applications, but their manual construction consumes time and personnel, and frequently results in low coverage. One alternative is the automatic construction of large resources from corpora like distributional thesauri, containing semantically associated words. However, as they may contain noise, there is a strong need for automatic ways of evaluating the quality of the resulting resource. This paper introduces a gold standard that can aid in this task. The BabelNet-Based Semantic Gold Standard (B2SG) was automatically constructed based on BabelNet and partly evaluated by human judges. It consists of sets of tests that present one target word, one related word and three unrelated words. B2SG contains 2,875 validated relations: 800 for verbs and 2,075 for nouns; these relations are divided among synonymy, antonymy and hypernymy. They can be used as the basis for evaluating the accuracy of the similarity relations on distributional thesauri by comparing the proximity of the target word with the related and unrelated options and observing if the related word has the highest similarity value among them. As a case study two distributional thesauri were also developed: one using surface forms from a large (1.5 billion word) corpus and the other using lemmatized forms from a smaller (409 million word) corpus. Both distributional thesauri were then evaluated against B2SG, and the one using lemmatized forms performed slightly better. Keywords: distributional thesauri, gold standard, lexical resource 1. Introduction The importance of resources such as WordNet (Fellbaum, 1998), that represent semantic relations between words, can be measured by the number of initiatives dedicated to (re)producing them in other languages, such as the Eu- roWordNet1 (Vossen, 1998) and the Global WordNet As- sociation2 (Bond and Paik, 2012). These resources have been windely used in numerous NLP applications, such as systems for Q&A, text simplification, and sentiment analy- sis. For Portuguese, some such initiatives include Onto.PT3 (Gonc¸alo Oliveira and Gomes, 2010), OpenWN-PT4 (de Paiva et al., 2012), MultiWordnet of Portuguese5, Word- Net.PT6 (Marrafa, 2002), WordNet.Br7 (Dias-da-Silva et al., 2008). However, the manual construction of this type of resource is costly and much time-consuming, in addition to having low coverage and being applicable to only one domain. More- over, its availability is limited or non-existent for many lan- guages. A popular alternative is the automatic construc- tion of distributional thesauri from corpora, resulting in a resource with semantic association among words. These techniques are language independent and applicable to any domain (Lin, 1998). As a consequence there is much atten- tion being devoted to the systematic construction, evalua- tion and enhancement of distributional thesauri. The automatic evaluation of such thesauri, in particular, is 1http://www.illc.uva.nl/EuroWordNet/ 2http://globalwordnet.org/ wordnets-in-the-world/ 3http://ontopt.dei.uc.pt 4https://github.com/arademaker/ openWordnet-PT 5http://mwnpt.di.fc.ul.pt/ 6http://www.clul.ul.pt/clg/wordnetpt/ index.html 7http://143.107.183.175:21380/wordnetbr a complex task, because of the lack of resources with in- formation about the similarity between words. Moreover, due to the large …

Content truncated to 500 words