Behavioral Factors in Interactive Training of Text Classifiers

Behavioral Factors in Interactive Training of Text Classifiers Burr Settles Machine Learning Department Carnegie Mellon University Pittsburgh PA 15213, USA bsettles@cs.cmu.edu Xiaojin Zhu Computer Sciences Department University of Wisconsin Madison WI 53715, USA jerryzhu@cs.wisc.edu Abstract This paper describes a user study where hu- mans interactively train automatic text clas- sifiers. We attempt to replicate previous re- sults using multiple “average” Internet users instead of a few domain experts as annotators. We also analyze user annotation behaviors to find that certain labeling actions have an im- pact on classifier accuracy, drawing attention to the important role these behavioral factors play in interactive learning systems. 1 Introduction There is growing interest in methods that incorpo- rate human domain knowledge in machine learning algorithms, either as priors on model parameters or as constraints in an objective function. Such ap- proaches lend themselves well to natural language tasks, where input features are often discrete vari- ables that carry semantic meaning (e.g., words). A feature label is a simple but expressive form of do- main knowledge that has received considerable at- tention recently (Druck et al., 2008; Melville et al., 2009). For example, a single feature (word) can be used to indicate a particular label or set of labels, such as “excellent” ⇒ positive or “terrible” ⇒ neg- ative, which might be useful word-label rules for a sentiment analysis task. Contemporary work has also focused on mak- ing such learning algorithms active, by enabling them to pose “queries” in the form of feature-based rules to be labeled by annotators in addition to — and sometimes lieu of — data instances such as documents (Attenberg et al., 2010; Druck et al., 2009). These concepts were recently implemented in a practical system for interactive training of text classifiers called DUALIST1. Settles (2011) reports that, in user experiments with real annotators, hu- mans were able to train near state of the art classi- fiers with only a few minutes of effort. However, there were only five subjects, who were all com- puter science researchers. It is possible that these positive results can be attributed to the subjects’ im- plicit familiarity with machine learning and natural language processing algorithms. This short paper sheds more light on previous ex- periments by replicating them with many more hu- man subjects, and of a different type: non-experts recruited through the Amazon Mechanical Turk ser- vice2. We also analyze the impact of annotator be- havior on the resulting classifiers, and suggest rela- tionships to recent work in curriculum learning. 2 DUALIST Figure 1 shows a screenshot of DUALIST, an inter- active machine learning system for quickly build- ing text classifiers. The annotator is allowed to take three kinds of actions: Œ label query docu- ments (instances) by clicking class-label buttons in the left panel,  label query words (features) by selecting them from the class-label columns in the right panel, or Ž “volunteer” domain knowledge by typing labeled words into a text box at the top of each class column. The underlying classifier …

Content truncated to 500 words