Using Crowdsourcing to get Representations based on Regular Expressions

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1476–1480, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Using crowdsourcing to get representations based on regular expressions Anders Søgaard and Hector Martinez and Jakob Elming and Anders Johannsen Center for Language Technology University of Copenhagen DK-2300 Copenhagen S {soegaard|alonso|zmk867|ajohannsen}@hum.ku.dk Abstract Often the bottleneck in document classifica- tion is finding good representations that zoom in on the most important aspects of the doc- uments. Most research uses n-gram repre- sentations, but relevant features often occur discontinuously, e.g., not...good in sentiment analysis. In this paper we present experi- ments getting experts to provide regular ex- pressions, as well as crowdsourced annota- tion tasks from which regular expressions can be derived. Somewhat surprisingly, it turns out that these crowdsourced feature combina- tions outperform automatic feature combina- tion methods, as well as expert features, by a very large margin and reduce error by 24-41% over n-gram representations. 1 Introduction Finding good representations of classification prob- lems is often glossed over in the literature. Sev- eral authors have emphasized the need to pay more attention to finding such representations (Wagstaff, 2012; Domingos, 2012), but in document classifica- tion most research still uses n-gram representations. This paper considers two document classification problems where such representations seem inade- quate. The problems are answer scoring (Burstein et al., 1998), on data from stackoverflow.com, and multi-attribute sentiment analysis (McAuley et al., 2012). We argue that in order to adequately repre- sent such problems we need discontinuous features, i.e., regular expressions. The problem with using regular expressions as features is of course that even with a finite vocab- ulary we can generate infinitely many regular ex- pressions that match our documents. We suggest to use expert knowledge or crowdsourcing in the loop. In particular we present experiments where standard representations are augmented with features from a few hours of manual work, by machine learning ex- perts or by turkers. Somewhat surprisingly, we find that features de- rived from crowdsourced annotation tasks lead to the best results across the three datasets. While crowd- sourcing of annotation tasks has become increasing popular in NLP, this is, to the best of our knowledge, the first attempt to crowdsource the problem of find- ing good representations. 1.1 Related work Musat et al. (2012) design a collaborative two-player game for sentiment annotation and collecting a sen- timent lexicon. One player guesses the sentiment of a text and picks a word from it that is representative of its sentiment. The other player also provides a guess observing only this word. If the two guesses agree, both players get a point. The idea of gam- ifying the problem of finding good representations goes beyond crowdsourcing, but is not considered here. Boyd-Graber et al. (2012) crowdsource the feature weighting problem, but using standard rep- resentations. The work most similar to ours is prob- ably Tamuz et al. (2011), who learn a ’crowd kernel’ by asking annotators to rate examples by similarity, …

Content truncated to 500 words