Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1294–1304, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose Daniele Vannella, David Jurgens, Daniele Scarfini, Domenico Toscani and Roberto Navigli Department of Computer Science Sapienza University of Rome surname@di.uniroma1.it Abstract Large-scale knowledge bases are impor- tant assets in NLP. Frequently, such re- sources are constructed through automatic mergers of complementary resources, such as WordNet and Wikipedia. However, manually validating these resources is pro- hibitively expensive, even when using methods such as crowdsourcing. We pro- pose a cost-effective method of validat- ing and extending knowledge bases using video games with a purpose. Two video games were created to validate concept- concept and concept-image relations. In experiments comparing with crowdsourc- ing, we show that video game-based vali- dation consistently leads to higher-quality annotations, even when players are not compensated. 1 Introduction Large-scale knowledge bases are an essential component of many approaches in Natural Lan- guage Processing (NLP). Semantic knowledge bases such as WordNet (Fellbaum, 1998), YAGO (Suchanek et al., 2007), and BabelNet (Navigli and Ponzetto, 2010) provide ontological struc- ture that enables a wide range of tasks, such as measuring semantic relatedness (Budanitsky and Hirst, 2006) and similarity (Pilehvar et al., 2013), paraphrasing (Kauchak and Barzilay, 2006), and word sense disambiguation (Navigli and Ponzetto, 2012; Moro et al., 2014). Furthermore, such knowledge bases are essential for building unsu- pervised algorithms when training data is sparse or unavailable. However, constructing and updat- ing semantic knowledge bases is often limited by the significant time and human resources required. Recent approaches have attempted to build or extend these knowledge bases automatically. For example, Snow et al. (2006) and Navigli (2005) extend WordNet using distributional or structural features to identify novel semantic connections between concepts. The recent advent of large semi-structured resources has enabled the creation of new semantic knowledge bases (Medelyan et al., 2009; Hovy et al., 2013) through automati- cally merging WordNet and Wikipedia (Suchanek et al., 2007; Navigli and Ponzetto, 2010; Nie- mann and Gurevych, 2011). While these auto- matic approaches offer the scale needed for open- domain applications, the automatic processes of- ten introduce errors, which can prove detrimental to downstream applications. To overcome issues from fully-automatic construction methods, sev- eral works have proposed validating or extending knowledge bases using crowdsourcing (Biemann and Nygaard, 2010; Eom et al., 2012; Sarasua et al., 2012). However, these methods, too, are lim- ited by the resources required for acquiring large numbers of responses. In this paper, we propose validating and extend- ing semantic knowledge bases using video games with a purpose. Here, the annotation tasks are transformed into elements of a video game where players accomplish their jobs by virtue of playing the game, rather than by performing a more tradi- tional annotation task. While prior efforts in NLP have incorporated games for performing annota- tion and validation (Siorpaes and Hepp, 2008b; Herda˘gdelen and Baroni, 2012; Poesio …

Content truncated to 500 words