Team Taurus at SemEval-2019 Task 9: Expert-informed pattern recognition for suggestion mining

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1247–1253 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1247 Team Taurus at SemEval-2019 Task 9: Expert-informed pattern recognition for suggestion mining Nelleke Oostdijk Centre for Language Studies Radboud University Nijmegen, The Netherlands N.Oostdijk@let.ru.nl Hans van Halteren Centre for Language Studies Radboud University Nijmegen, The Netherlands hvh@let.ru.nl Abstract This paper presents our submissions to SemEval-2019 Task9, Suggestion Mining. Our system is one in a series of systems in which we compare an approach using expert- defined rules with a comparable one using ma- chine learning. We target tasks with a syntac- tic or semantic component that might be better described by a human understanding the task than by a machine learner only able to count features. For Semeval-2019 Task 9, the expert rules clearly outperformed our machine learn- ing model when training and testing on equally balanced testsets. 1 Introduction In the field of natural language processing, ap- proaches featuring machine learning (ML) nowa- days predominate. These have been shown to be quite effective with a wide range of tasks, in- cluding text mining, authorship attribution, and text classification. They are particularly suited for dealing with large data volumes and are ro- bust in the sense that they can handle quite ‘noisy’ data. Unlike expert-informed approaches where rules need to be hand-crafted and apply only under predefined conditions, ML approaches learn from training data and are able to extrapolate. Propo- nents of ML approaches tend to dismiss the en- terprise of hand-crafting rules as difficult, error- prone, time-consuming, and generally ineffective as even an extensive set of complex rules is bound to be incomplete and difficult to maintain. Over the past years we have conducted a num- ber of studies directed at the extraction of ac- tionable information from microblogs. These in- clude a range of topic areas and domains, in- cluding the detection of threatening tweets (Oost- dijk and van Halteren, 2013a,b), the identifica- tion of potentially contaminated food supplements in forum posts (Oostdijk et al., submitted), and topic/event detection in tweets about natural dis- asters (floods (H¨urriyeto˘glu et al., 2016), earth- quakes (H¨urriyeto˘glu and Oostdijk, 2017; Oost- dijk and H¨urriyeto˘glu, 2017)), about traffic flow (Oostdijk et al., 2016) and about outbreaks of the flu (H¨urriyeto˘glu et al., 2017). In each case we have been exploring the role of the human ex- pert in an expert-informed pattern recognition ap- proach and a comparable ML approach, seeking out the strengths and weaknesses of either and at- tempting to arrive at a superior hybrid approach. Semeval-2019, Task 9, Suggestion Mining (Negi et al., 2019), appeared to be another task that would lend itself to human rule building. As sug- gestions may be phrased in many different ways, successfully recognising that an utterance contains a suggestion requires human understanding of the context. Also, the amount of available training data was quite limited so that bringing into play the human expert’s knowledge of the forms that suggestions may take …

Content truncated to 500 words