INRIA at SemEval-2019 Task 9: Suggestion Mining Using SVM with Handcrafted Features

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1204–1207 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1204 INRIA at SemEval-2019 Task 9: Suggestion Mining Using SVM with Handcrafted Features Ilia Markov INRIA Paris, France ilia.markov@inria.fr Eric Villemonte De la Clergerie INRIA Paris, France eric.de la clergerie@inria.fr Abstract We present the INRIA approach to the sugges- tion mining task at SemEval 2019. The task consists of two subtasks: suggestion mining under single-domain (Subtask A) and cross- domain (Subtask B) settings. We used the Support Vector Machines algorithm trained on handcrafted features, function words, senti- ment features, digits, and verbs for Subtask A, and handcrafted features for Subtask B. Our best run archived a F1-score of 51.18% on Subtask A, and ranked in the top ten of the sub- missions for Subtask B with 73.30% F1-score. 1 Introduction Suggestion mining can be viewed as a task of ex- tracting suggestions from unstructured text sam- ples (Ramanand et al., 2010; Negi and Buitelaar, 2015). The task goes beyond the sentiment polar- ity detection and is useful for a variety of purposes, e.g., organizations can improve their products bas- ing on the suggestions from online sources without the need of manually analyzing large amounts of unstructured data (Dong et al., 2013). In this first edition of the suggestion mining SemEval task (Negi et al., 2019), two settings of the task are addressed: single-domain (or domain- specific) suggestion mining, where the training, development, and test sets belong to the same do- main (in the context of this shared task, sugges- tion forum for Windows platform developers), and cross-domain setting, where training and develop- ment/test sets belong to different domains (train- ing on developer suggestion forums and testing on hotel reviews). In the both domains, only explicit expressions of suggestions are considered: lexical cues of a suggestion are explicitly mentioned in the text (Negi et al., 2018). We approach the task from a machine-learning perspective as a binary classification of given sentences into suggestion and non-suggestion classes. We propose a straightforward approach that can be applied when the availability of train- ing/evaluation data and external linguistic re- sources is scarce, and evaluate it in the context of this shared task. We were particularly interested in evaluating our approach under cross-domain con- ditions (Subtask B), since this setting is more com- mon in a real-word scenario of the task. Further, we briefly describe the datasets used in the competition and focus on the configuration of our system. 2 Data The training dataset provided by the organizers, as well as the development and test sets for Subtask A consist of explicit suggestion and non-suggestion sentences extracted from the feedback posts on the Universal Windows Platform1, while the de- velopment and test sets for Subtask B are on a different domain: a subset of the sentiment anal- ysis dataset of hotel reviews from the TripAdvisor website (Wachsmuth et al., 2014). The training, development (dev.), and test datasets statistics in terms of the …

Content truncated to 500 words