SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 877–887 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 877 SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums Sapna Negi Genesys Telecommunication Laboratories Galway, Ireland sapna.negi1 @gmail.com Tobias Daudert National University of Ireland Galway, Ireland tobias.daudert @insight-centre.org Paul Buitelaar National University of Ireland Galway, Ireland paul.buitelaar @insight-centre.org Abstract We present the pilot SemEval task on Sugges- tion Mining. The task consists of subtasks A and B, where we created labeled data from feedback forum and hotel reviews respectively. Subtask A provides training and test data from the same domain, while Subtask B evaluates the system on a test dataset from a differ- ent domain than the available training data. 33 teams participated in the shared task, with a total of 50 members. We summarize the problem definition, benchmark dataset prepa- ration, and methods used by the participating teams, providing details of the methods used by the top ranked systems. The dataset is made freely available to help advance the research in suggestion mining, and reproduce the systems submitted under this task. 1 Introduction State of the art opinion mining systems provide numerical summaries of sentiments and tend to overlook additional descriptive and potentially useful content present in the opinionated text. We stress that such content also encompass information like suggestions, tips, and advice, which is otherwise explicitly sought by the stakeholders. For example, hotel reviews often contain room tips, i.e., which room should be preferred in a hotel. Likewise, tips on restaurants, shops, sightseeing, etc. are also present within the hotel reviews. On the other hand, platforms like Tripadvisor1, which collect hotel and restaurant related opinions, request the reviewers to fill up the room tips section in addition to the hotel review. Likewise, sentences expressing advice, tips, and recommendations relating to a target entity can often be present in text available from different types of data sources, like blogs, microblogs, discussions, etc. Such sentences can 1https://www.tripadvisor.com be collectively referred to as suggestions. With the increasing availability of opinionated text, methods for automatic detection of suggestions can be employed for different use cases. Some example use cases are the extraction of sugges- tions for brand improvement, the extraction of tips and advice for customers, the extraction of the expressions of recommendations from unstruc- tured data in order to aid recommender systems, or the summarisation of suggestion forums where suggestion providers often tend to provide context in their responses (Figure 1) which gets repetitive over a large number of responses relating to the same entity. The task of automatic identification of suggestions in a given text is referred to as suggestion mining (Brun and Hagege, 2013). Studies performed on suggestion mining have defined it as a sentence classification task, where class prediction has to be made on each sentence of a given text, classes being suggestion and non suggestion (Negi, 2016). State of the art opinion mining systems have mostly focused on identi- fying sentiment polarity …

Content truncated to 500 words