ZQM at SemEval-2019 Task9: A Single Layer CNN Based on Pre-trained Model for Suggestion Mining

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1287–1291 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1287 ZQM at SemEval-2019 Task9: A Single Layer CNN Based on Pre-trained Model for Suggestion Mining Qimin Zhou, Zhengxin Zhang, Hao Wu*, Linmao Wang School of Information Science and Engineering, Yunnan University Chenggong Campus, Kunming, P.R. China {zqmynu,zzxynu}@gmail.com, haowu@ynu.edu.cn, wlmdyx@gmail.com Abstract This paper describes our system that competed at SemEval 2019 Task 9 - SubTask A: ”Sug- gestion Mining from Online Reviews and Fo- rums”. Our system fuses the convolutional neural network and the latest BERT model to conduct suggestion mining. In our system, the input of convolutional neural network is the embedding vectors which are drawn from the pre-trained BERT model. And to enhance the effectiveness of the whole system, the pre- trained BERT model is fine-tuned by provid- ed datasets before the procedure of embedding vectors extraction. Empirical results show the effectiveness of our model which obtained 9th position out of 34 teams with F1 score equals to 0.715. 1 Introduction Suggestion mining is defined as the extraction of suggestions from unstructured text (Negi et al., 2018). Suggestion mining is still a relatively y- oung research area as compared to other natural language processing issues like sentiment analy- sis (Negi and Buitelaar, 2015). While suggestion mining is of great commercial value for organi- sations to improve the quality of their entities by considering the positive and negative opinions col- lected from platforms. The target of this task is to automatically classify the sentences collected from online reviews and forums into two classes which are suggestion and non-suggestion respec- tively (Negi et al., 2019). BERT which stands for Bidirectional Encoder Representation from Transformers is the latest breakthrough in the field of NLP provided by Google Research (Devlin et al., 2018). It has substantially advanced the state-of-the-art in many NLP tasks, especially in question answering (Al- berti et al., 2019). More importantly, it provides a widely applicable tool for representation learning which can be generalized to many NLP tasks. For this subtask, we firstly learn the word or sentence embeddings utilizing the pre-trained BERT model. Then the embedding vectors are extracted from BERT as the input of the subse- quent model. It is worth noting that we have fine- tuned the pre-trained BERT model with provided dataset before the embedding vectors are extract- ed. In other words, this part is equivalent to the conventional embedding layer. This strategy is a little bit like ELMO (Peters et al., 2018). As for the upper layer of this system, convolutional neu- ral network (CNN) is adopted herein to process the features. Although CNN is originally invent- ed for tackling computer vision issues, while it has subsequently been shown to be effective for many NLP tasks (Kim, 2014; Zhang and Wallace, 2015; Dong et al., 2015). The remainder of the paper is organized as fol- lows. Section 2 describes the detailed architecture of our system. Section 3 reports the experimental results …

Content truncated to 500 words