Semi-Supervised Cause Identification from Aviation Safety Reports

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 843–851, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Semi-Supervised Cause Identification from Aviation Safety Reports Isaac Persing and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,vince}@hlt.utdallas.edu Abstract We introduce cause identification, a new problem involving classification of in- cident reports in the aviation domain. Specifically, given a set of pre-defined causes, a cause identification system seeks to identify all and only those causes that can explain why the aviation incident de- scribed in a given report occurred. The dif- ficulty of cause identification stems in part from the fact that it is a multi-class, multi- label categorization task, and in part from the skewness of the class distributions and the scarcity of annotated reports. To im- prove the performance of a cause identi- fication system for the minority classes, we present a bootstrapping algorithm that automatically augments a training set by learning from a small amount of labeled data and a large amount of unlabeled data. Experimental results show that our algo- rithm yields a relative error reduction of 6.3% in F-measure for the minority classes in comparison to a baseline that learns solely from the labeled data. 1 Introduction Automatic text classification is one of the most im- portant applications in natural language process- ing (NLP). The difficulty of a text classification task depends on various factors, but typically, the task can be difficult if (1) the amount of labeled data available for learning the task is small; (2) it involves multiple classes; (3) it involves multi- label categorization, where more than one label can be assigned to each document; (4) the class distributions are skewed, with some categories significantly outnumbering the others; and (5) the documents belong to the same domain (e.g., movie review classification). In particular, when the doc- uments to be classified are from the same domain, they tend to be more similar to each other with respect to word usage, thus making the classes less easily separable. This is one of the reasons why topic-based classification, even with multiple classes as in the 20 Newsgroups dataset1, tends to be easier than review classification, where reviews from the same domain are to be classified accord- ing to the sentiment expressed2. In this paper, we introduce a new text classifi- cation problem involving the Aviation Safety Re- porting System (ASRS) that can be viewed as a difficult task along each of the five dimensions dis- cussed above. Established in 1967, ASRS collects voluntarily submitted reports about aviation safety incidents written by flight crews, attendants, con- trollers, and other related parties. These incident reports are made publicly available to researchers for automatic analysis, with the ultimate goal of improving the aviation safety situation. One cen- tral task in the automatic analysis of these reports is cause identification, or the identification of why an incident happened. Aviation safety experts at NASA have …

Content truncated to 500 words