Using “Annotator Rationales” to Improve Machine Learning for Text Categorization

Proceedings of NAACL HLT 2007, pages 260–267, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Using “Annotator Rationales” to Improve Machine Learning for Text Categorization∗ Omar F. Zaidan and Jason Eisner Department of Computer Science Johns Hopkins University Baltimore, MD 21218, USA {ozaidan,jason}@cs.jhu.edu Christine D. Piatko JHU Applied Physics Laboratory 11100 Johns Hopkins Road Laurel, MD 20723 USA christine.piatko@jhuapl.edu Abstract We propose a new framework for supervised ma- chine learning. Our goal is to learn from smaller amounts of supervised training data, by collecting a richer kind of training data: annotations with “ra- tionales.” When annotating an example, the hu- man teacher will also highlight evidence support- ing this annotation—thereby teaching the machine learner why the example belongs to the category. We provide some rationale-annotated data and present a learning method that exploits the rationales during training to boost performance significantly on a sam- ple task, namely sentiment classification of movie reviews. We hypothesize that in some situations, providing rationales is a more fruitful use of an an- notator’s time than annotating more examples. 1 Introduction Annotation cost is a bottleneck for many natural lan- guage processing applications. While supervised machine learning systems are effective, it is labor- intensive and expensive to construct the many train- ing examples needed. Previous research has ex- plored active or semi-supervised learning as possible ways to lessen this burden. We propose a new way of breaking this annotation bottleneck. Annotators currently indicate what the correct answers are on training data. We propose that they should also indicate why, at least by coarse hints. We suggest new machine learning approaches that can benefit from this “why” information. For example, an annotator who is categorizing phrases or documents might also be asked to high- light a few substrings that significantly influenced her judgment. We call such clues “rationales.” They need not correspond to machine learning features. ∗This work was supported by the JHU WSE/APL Partner- ship Fund; National Science Foundation grant No. 0347822 to the second author; and an APL Hafstad Fellowship to the third. In some circumstances, rationales should not be too expensive or time-consuming to collect. As long as the annotator is spending the time to study exam- ple xi and classify it, it may not require much extra effort for her to mark reasons for her classification. 2 Using Rationales to Aid Learning We will not rely exclusively on the rationales, but use them only as an added source of information. The idea is to help direct the learning algorithm’s attention—helping it tease apart signal from noise. Machine learning algorithms face a well-known “credit assignment” problem. Given a complex da- tum xi and the desired response yi, many features of xi could be responsible for the choice of yi. The learning algorithm must tease out which features were actually responsible. This requires a lot of training data, and often a lot of computation as well. Our rationales offer a shortcut to solving this “credit assignment” problem, by providing the learning algorithm with …

Content truncated to 500 words