Online Gaming for Crowd-sourcing Phrase-equivalents

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1238–1247, Dublin, Ireland, August 23-29 2014. Online Gaming for Crowd-sourcing Phrase-equivalents A. Kumaran Microsoft Research Bangalore, India a.kumaran@microsoft.com Melissa Densmore University of Cape Town Cape Town, South Africa mdensmore@acm.org Shaishav Kumar Microsoft Research Bangalore, India v-shaisk@microsoft.com Abstract We propose the use of a game with a purpose (GWAP) to facilitate crowd-sourcing of phrase- equivalents, as an alternative to expert or paid crowd-sourcing. Doodling is an online multi- player game, in which one player (drawer), draws pictures on a shared board to get the other players (guessers) to guess the meaning behind an assigned phrase. In this paper we describe the system and results from several experiments intended to improve the quality of information generated by the play. In addition, we describe the mechanism by which we take candidate phrases generated during the games and filter out true phrase equivalents. We expect that, at scale, this game will be more cost-efficient than paid mechanisms for a similar task, and demon- strate this by comparing the productivity of an hour of game play to an equivalent crowd-sourced Amazon Mechanical Turk task to produce phrase-equivalents over one week. 1 Introduction While it is fairly well known when individual words have the same meaning, it is far more difficult to determine when phrases or even sentences carry the same basic idea. While it might be possible to address this task with machine learning techniques, building a corpus of sentences from which to seed a database requires human intelligence. We suggest a game with a purpose (GWAP) that will serve to generate phrases with similar meanings, while simultaneously providing meta-information about the quality of the match. In this drawing game, called Doodling, individuals compete in groups to guess the meaning behind a given drawing that is being drawn by one designated drawer trying to convey a given phrase or a short sentence. The designated drawer decides when a guessed phrase matches the source phrase. For example “How far is the airport?” might match semantically “What is the distance to the airport?” In addition, the drawer can indicate for each partial guess how close it is on a scale of 1-3 to help the guessers converge on phrases that will match the given phrase or sentence. We then pass all of the guesses and annotations through an SVM classifier to automatically identify potential phrase-equiv- alents. In this study we examine several techniques for using this system to generate high quality data while also making the game more enjoyable. We measure the efficacy of each technique by comparing our results to a gold standard: using human evaluators to rate the phrase matches generated through the game manually. We also compare Doodling to a paid crowdsourcing paradigm – Amazon’s Mechanical Turk – to source phrase equivalents for the same set of phrases, and we show that our approach might be cost effective for large scale sourcing of paraphrases of equivalent quality. 2 Background In …

Content truncated to 500 words