More Features Are Not Always Better: Evaluating Generalizing Models in Incident Type Classification of Tweets

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 421–430, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. More Features Are Not Always Better: Evaluating Generalizing Models in Incident Type Classification of Tweets Axel Schulz Business Intelligence Marketing DB Fernverkehr AG Germany schulz.axel@gmx.net Christian Guckelsberger Computational Creativity Group Goldsmiths College, University of London United Kingdom c.guckelsberger@gold.ac.uk Benedikt Schmidt Telecooperation Lab, Technische Universit¨at Darmstadt, Germany benedikt.schmidt@tk.informatik.tu-darmstadt.de Abstract Social media represents a rich source of up- to-date information about events such as in- cidents. The sheer amount of available infor- mation makes machine learning approaches a necessity for further processing. This learn- ing problem is often concerned with region- ally restricted datasets such as data from only one city. Because social media data such as tweets varies considerably across differ- ent cities, the training of efficient models re- quires labeling data from each city of inter- est, which is costly and time consuming. In this study, we investigate which features are most suitable for training generalizable models, i.e., models that show good per- formance across different datasets. We re- implemented the most popular features from the state of the art in addition to other novel approaches, and evaluated them on data from ten different cities. We show that many so- phisticated features are not necessarily valu- able for training a generalized model and are outperformed by classic features such as plain word-n-grams and character-n-grams. 1 Introduction Incident information contained in social media has proven to frequently include information not cap- tured by standard emergency channels (e.g. 911 calls, bystander reports). Therefore, stakeholders like emergency management and city administra- tion can highly benefit from social media. Due to its unstructured and unfocused nature, automatic filtering of social media content is a necessity for further analysis. A standard approach for this fil- tering is automatic classification using a trained machine learning model (Agarwal et al., 2012; Schulz et al., 2013; Schulz et al., 2015b). A problem for the classification approach is that language, style and named entities used in social media highly vary across different regions. Con- sider the following two tweets as examples: “RT: @People 0noe friday afternoon in heavy traffic, car crash on I-90, right lane closed” and “Road blocked due to traffic collision on I-495”. Both tweets comprise entities that might refer to the same thing with different wording, either on a se- mantically low (“accident” and “car collision”) or more abstract level (“I90” and “I-495”). With simple syntactical text similarity approaches using standard bag of words features, it is not easily pos- sible to make use of this semantic similarity, even though it is highly valuable for classification. These limitations impose constraints on the dataset, because tokens are likely to be related to the location where the text was created or con- tain location- or incident-sensitive topics. Models trained using spatially and temporally restricted data from one region are bound by the specific as- pects of language and style expressed in the train- ing …

Content truncated to 500 words