A Simpler and More Generalizable Story Detector using Verb and Character Features

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2708–2715 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Simpler and More Generalizable Story Detector using Verb and Character Features Joshua D. Eisenberg and Mark A. Finlayson 11200 S.W. 8th Street, ECS Building, Miami, FL 33141 School of Computing and Information Sciences Florida International University Miami, Florida {jeise003, markaf}@fiu.edu Abstract Story detection is the task of determining whether or not a unit of text contains a story. Prior approaches achieved a max- imum performance of 0.66 F1, and did not generalize well across different cor- pora. We present a new state-of-the-art detector that achieves a maximum per- formance of 0.75 F1 (a 14% improve- ment), with significantly greater general- izability than previous work. In partic- ular, our detector achieves performance above 0.70 F1 across a variety of combi- nations of lexically different corpora for training and testing, as well as dramatic improvements (up to 4,000%) in perfor- mance when trained on a small, disfluent data set. The new detector uses two basic types of features–ones related to events, and ones related to characters–totaling 283 specific features overall; previous detec- tors used tens of thousands of features, and so this detector represents a significant simplification along with increased perfor- mance. 1 Motivation Understanding stories is a long-held goal of both artificial intelligence and natural language pro- cessing. Stories can be used for many interest- ing natural language processing tasks, and much can be learned from them, including concrete facts about specific events, people, and things; com- monsense knowledge about the world; and cul- tural knowledge about the societies in which we live. Applying NLP directly to the large and growing number of stories available electronically, however, has been limited by our inability to effi- ciently separate story from non-story text. For the most part, studies of stories per se has relied on manual curation of story data sets (Mostafazadeh et al., 2016), which is, naturally, time-consuming, expensive, and doesn’t scale. These human-driven methods pay no attention to the large number of stories generated daily in news, entertainment, and social media. The goal of this work is to build and evaluate a high performing story detector that is both sim- ple in design and generalizable across lexically different story corpora. Our definition of story can be found in §1.2, and is based on definitions used in prior work on story detection. Previous approaches to story detection have relied on tens of thousands of features (Ceran et al., 2012; Gor- don and Swanson, 2009), and have used compli- cated pre-processing pipelines (Ceran et al., 2012). Moreover these prior systems, while clearly im- portant advances, did not, arguably, include fea- tures that captured the “essence” of stories. Fur- thermore, these prior efforts had poor generaliz- ability, i.e. when trained on one corpus, the detec- tors perform poorly when tested on a different cor- pus. Building on this prior work, we begin to ad- dress these shortcomings, presenting a …

Content truncated to 500 words