OleNet at SemEval-2019 Task 9: BERT based Multi-Perspective Models for Suggestion Mining

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1231–1236 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1231 OleNet at SemEval-2019 Task 9: BERT based Multi-Perspective Models for Suggestion Mining Jiaxiang Liu, Shuohuan Wang, and Yu Sun Baidu Inc., Beijing, China, {liujiaxiang, wangshuohuan, sunyu02}@baidu.com Abstract This paper describes our system participated in Task 9 of SemEval-2019: the task is fo- cused on suggestion mining and it aims to classify given sentences into suggestion and non-suggestion classes in domain specific and cross domain training setting respectively. We propose a multi-perspective architecture for learning representations by using different classical models including Convolutional Neu- ral Networks (CNN), Gated Recurrent Units (GRU), Feed Forward Attention (FFA), etc. To leverage the semantics distributed in large amount of unsupervised data, we also have adopted the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model as an encoder to produce sentence and word representations. The proposed architec- ture is applied for both sub-tasks, and achieved f1-score of 0.7812 for subtask A, and 0.8579 for subtask B. We won the first and second place for the two tasks respectively in the fi- nal competition. 1 Introduction Suggestion mining, which can be defined as the extraction of suggestions from unstructured text, where the term suggestions refers to the expres- sions of tips, advice, recommendations etc. (Negi et al., 2018). For example, I would recommend doing the upgrade to be sure you have the best chance at trouble free operation. and Be sure to specify a room at the back of the hotel. should be a suggestion for electronics and hotel separately. Collecting suggestions is an integral step of any decision making process. A suggestion mining system could extract exact suggestion sentences from a retrieved document, which would enable the user to collect suggestions from a much larger number of pages than they could manually read over a short span of time. Suggestion mining remains a relatively young area. So far, it has usually been defined as a problem of classifying sentences of a given text into suggestion and non-suggestion classes. Mostly rule-based systems have so far been de- veloped, and very few statistical classifiers have been proposed (Negi and Buitelaar, 2017) (Negi et al., 2016) (Negi and Buitelaar, 2015) (Brun and Hag`ege, 2013). A related field to suggestion min- ing is sentiment classification which given a sen- tence or a document, it should infer the sentiment polarity e.g. positive, negative, neutral. So, many classical sentiment classification systems can be used in suggestion mining like the widely used CNN-based models (Kim, 2014) or RNN-based models (Kawakami, 2008). However, there are still many challenges in this suggestion mining task. First of all, both of the subtasks suffers from severely lack of data. Second, one of the sub- tasks requires the model should have transferabil- ity without seeing any of the target domain data. To tackle those problems, knowledge transfer or transfer learning between domains would be desir- able. In recent years, transfer learning techniques have been …

Content truncated to 500 words