Online Word Games for Semantic Data Collection

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Online Word Games for Semantic Data Collection David Vickrey Aaron Bronzan William Choi Aman Kumar Jason Turner-Maier Arthur Wang Daphne Koller Stanford University Stanford, CA 94305-9010 {dvickrey,abronzan,aman,arthurex,koller}@cs.stanford.edu {wchoi25,jasonptm}@stanford.edu Abstract Obtaining labeled data is a significant obstacle for many NLP tasks. Recently, online games have been proposed as a new way of obtain- ing labeled data; games attract users by be- ing fun to play. In this paper, we consider the application of this idea to collecting seman- tic relations between words, such as hyper- nym/hyponym relationships. We built three online games, inspired by the real-life games of ScattergoriesTM and TabooTM. As of June 2008, players have entered nearly 800,000 data instances, in two categories. The first type of data consists of category/answer pairs (“Types of vehicle”,“car”), while the second is essentially free association data (“subma- rine”,”underwater”). We analyze both types of data in detail and discuss potential uses of the data. We show that we can extract from our data set a significant number of new hy- pernym/hyponym pairs not already found in WordNet. 1 Introduction One of the main difficulties in natural language pro- cessing is the lack of labeled data. Typically, obtain- ing labeled data requires hiring human annotators. Recently, building online games has been suggested an alternative to hiring annotators. For example, von Ahn and Dabbish (2004) built the ESP Game1, an online game in which players tag images with words that describe them. It is well known that there are large numbers of web users who will play online games. If a game is fun, there is a good chance that sufficiently many online users will play. We have several objectives in this paper. The first is to discuss design decisions in building word games for collecting data, and the effects of these decisions. The second is to describe the word games 1www.gwap.com/gwap/gamesPreview/espgame that we implemented and the kinds of data they are designed to collect. As of June 2008, our games have been online for nearly a year, and have col- lected nearly 800,000 data instances. The third goal is to analyze the resulting data and demonstrate that the data collected from our games is potentially use- ful in linguistic applications. As an example appli- cation, we show that the data we have collected can be used to augment WordNet (Fellbaum, 1998) with a significant number of new hypernyms. 2 General Design Guidelines Our primary goal is to produce a large amount of clean, useful data. Each of these three objectives (“large”, “clean”, and “useful”) has important im- plications for the design of our games. First, in order to collect large amounts of data, the game must be attractive to users. If the game is not fun, people will not play it. This requirement is perhaps the most significant factor to take into ac- count when designing a game. For one thing, it …

Content truncated to 500 words