It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation David Jurgens and Roberto Navigli Department of Computer Science Sapienza University of Rome {jurgens,navigli}@di.uniroma1.it Abstract Annotated data is prerequisite for many NLP applications. Acquiring large-scale annotated corpora is a major bottleneck, requiring sig- nificant time and resources. Recent work has proposed turning annotation into a game to increase its appeal and lower its cost; how- ever, current games are largely text-based and closely resemble traditional annotation tasks. We propose a new linguistic annota- tion paradigm that produces annotations from playing graphical video games. The effec- tiveness of this design is demonstrated using two video games: one to create a mapping from WordNet senses to images, and a sec- ond game that performs Word Sense Disam- biguation. Both games produce accurate re- sults. The first game yields annotation qual- ity equal to that of experts and a cost reduc- tion of 73% over equivalent crowdsourcing; the second game provides a 16.3% improve- ment in accuracy over current state-of-the-art sense disambiguation games with WordNet. 1 Introduction Nearly all of Natural Language Processing (NLP) depends on annotated examples, either for train- ing systems or for evaluating their quality. Typi- cally, annotations are created by linguistic experts or trained annotators. However, such effort is often very time- and cost-intensive, and as a result cre- ating large-scale annotated datasets remains a long- standing bottleneck for many areas of NLP. As an alternative to requiring expert-based anno- tations, many studies used untrained, online work- ers, commonly known as crowdsourcing. When successful, crowdsourcing enables gathering anno- tations at scale; however, its performance is still lim- ited by (1) the difficulty of expressing the annota- tion task as a simply-understood task suitable for the layman, (2) the cost of collecting many anno- tations, and (3) the tediousness of the task, which can fail to attract workers. Therefore, several groups have proposed an alternate annotation method us- ing games: an annotation task is converted into a game which, as a result of game play, produces an- notations (Pe-Than et al., 2012; Chamberlain et al., 2013). Turning an annotation task into a Game with a Purpose (GWAP) has been shown to lead to better quality results and higher worker engagement (Lee et al., 2013), thanks to the annotators being stimu- lated by the playful component. Furthermore, be- cause games may appeal to a different group of peo- ple than crowdsourcing, they provide a complemen- tary channel for attracting new annotators. Within NLP, gamified annotation tasks include anaphora resolution (Hladk´a et al., 2009; Poesio et al., 2013), paraphrasing (Chklovski and Gil, 2005), term associations (Artignan et al., 2009) and dis- ambiguation (Seemakurty et al., 2010; Venhuizen et al., 2013). The games’ interfaces typically incorpo- rate common game elements such as scores, leader- boards, or difficulty levels. However, the game it- self remains largely text-based, with a strong resem- blance to a traditional annotation task, and little re- semblance to games most people actively play. In …

Content truncated to 500 words