Community-Driven Crowdsourcing: Data Collection with Local Developers

Community-Driven Crowdsourcing: Data Collection with Local Developers Christina Funk, Michael Tseng, Ravindran Rajakumar, Linne Ha Google 1600 Amphitheatre Parkway, Mountain View, CA 94043 {christinafunk, michaeltseng, ravirajakumar, linne}@google.com Abstract We tested the viability of partnering with local developers to create custom annotation applications and to recruit and motivate crowd contributors from their communities to perform an annotation task consisting of the assignment of toxicity ratings to Wikipedia comments. We discuss the background of the project, the design of the community-driven approach, the developers’ execution of their applications and crowdsourcing programs, and the quantity, quality, and cost of judgments, in comparison with previous approaches. The community-driven approach resulted in local developers successfully creating four unique tools and collecting labeled data of sufficiently high quantity and quality. The creative approaches to the rating task presentation and crowdsourcing program design drew upon developers’ local knowledge of their own social networks, who also reported interest in the underlying problem that the data collection addresses. We consider the lessons that may be drawn from this project for implementing future iterations of the community-driven approach. Keywords: crowdsourcing, data diversity, sentiment annotation, tools and platforms 1. Introduction Labeled datasets for machine learning algorithms can con- tribute to robust and accurate models. Crowdsourcing presents one way to obtain more representative labeled datasets, but it is difficult to recruit crowd contributors from a diverse cross-section of communities. Furthermore, trans- actional crowdsourcing frameworks, where crowd contrib- utors are remote performers of an annotation task designed solely by the modelers of the problem, do not fully bene- fit from the potential insights of the crowd. In this paper, we present an approach for involving developer communi- ties in both annotation task development and crowdsourc- ing program design, so that they engage their local commu- nities to build labeled datasets that represent their world. Developers rooted in local communities are ideal interlop- ers to create tools that are effective for their particular con- text, as they can employ their community insights to build labeled datasets used to inform models that exemplify their locale and its people. While doing this, they can also con- tribute to a research space to which they might not normally have access, and as they learn, we learn with them. For the implementation of this approach, we chose a prob- lem in the sentiment annotation domain, where diverse judgments representative of several communities are valu- able due to the subjective nature of evaluating sentiment in a given context. Specifically, we chose to collect judg- ments on the toxicity of Wikipedia discussion comments, where a “toxic” comment is defined as any kind of hateful, aggressive, or disrespectful comment that is likely to make someone leave a discussion. The Conversation AI team at Jigsaw, a part of Alphabet, works on technology to promote civility in online discourse. They have shown that a classi- fier trained using data labeled through crowdsourcing can be as effective in identifying personal attacks as the aggre- gate work of three contributors (Wulczyn et al., 2017). Toxicity …

Content truncated to 500 words