Legal Docket Classification: Where Machine Learning Stumbles

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 438–446, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Legal Docket-Entry Classification: Where Machine Learning stumbles Ramesh Nallapati and Christopher D. Manning Natural Language Processing Group Department of Computer Science Stanford University Stanford, CA 94305 {nmramesh,manning}@cs.stanford.edu Abstract We investigate the problem of binary text clas- sification in the domain of legal docket entries. This work presents an illustrative instance of a domain-specific problem where the state- of-the-art Machine Learning (ML) classifiers such as SVMs are inadequate. Our investiga- tion into the reasons for the failure of these classifiers revealed two types of prominent er- rors which we call conjunctive and disjunctive errors. We developed simple heuristics to ad- dress one of these error types and improve the performance of the SVMs. Based on the in- tuition gained from our experiments, we also developed a simple propositional logic based classifier using hand-labeled features, that ad- dresses both types of errors simultaneously. We show that this new, but simple, approach outperforms all existing state-of-the-art ML models, with statistically significant gains. We hope this work serves as a motivating example of the need to build more expressive classifiers beyond the standard model classes, and to ad- dress text classification problems in such non- traditional domains. 1 Introduction Text Classification is a widely researched area, with publications spanning more than a decade (Yang and Liu, 1999). Although earlier models used logic based rules (Apt´e et al., 1994) and decision trees (Lewis and Ringuette, 1994), recently the emphasis has been on statistical classifiers such as the naive Bayes model (McCallum and Nigam, 1998), logis- tic regression (Zhang and Oles, 2001) and support vector machines (Joachims, 1998). Although several complex features were considered for classification, eventually researchers have settled down to simple bag-of-words features such as unigrams and some times bigrams (Dumais et al., 1998), thereby com- pletely ignoring the grammar and other semantic in- formation in the text. Despite this fact, the state- of-the-art performance is close to or above 90% on F1 scores on most standard test collections such as Reuters, 20 newsgroups, etc. (Bekkerman et al., 2003). As such, most researchers and practitioners believe text classification technology has reached a mature state, where it is suitable for deployment in real life applications. In this work, we present a text classification prob- lem from the legal domain which challenges some of our understanding of text classification problems. In the new domain, we found that the standard ML approaches using bag-of-words features perform rel- atively poorly. Not only that, we noticed that the linear form (or even polynomial form) used by these classifiers is inadequate to capture the semantics of the text. Our investigation into the shortcomings of the traditional models such as SVMs, lead us to build a simple propositional logic based classifier using hand-labeled features that outperforms these strong baselines. Although the new model by itself is interesting, the main objective of our work is to present the text classification …

Content truncated to 500 words