m_y at SemEval-2019 Task 9: Exploring BERT for Suggestion Mining

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 888–892 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 888 m y at SemEval-2019 Task 9: Exploring BERT for Suggestion Mining Masahiro Yamamoto Sony Corporation, Tokyo, Japan. Masahiro.A.Yamamoto@sony.com Toshiyuki Sekiya Sony Corporation, Tokyo, Japan. Toshiyuki.Sekiya@sony.com Abstract This paper presents our system to the SemEval-2019 Task 9, Suggestion Mining from Online Reviews and Forums. The goal of this task is to extract suggestions such as the expressions of tips, advice, and recommenda- tions. We explore Bidirectional Encoder Rep- resentations from Transformers (BERT) focus- ing on target domain pre-training in Subtask A which provides training and test datasets in the same domain. In Subtask B, the cross domain suggestion mining task, we apply the idea of distant supervision. Our system obtained the third place in Subtask A and the fifth place in Subtask B, which demonstrates its efficacy of our approaches. 1 Introduction In SemEval2019 Task 9, participants are required to build a model which can classify given sen- tences into suggestion and non-suggestion classes. We participate in two sub-tasks: domain specific suggestion mining task (Subtask A1) and cross do- main suggestion mining task (Subtask B2). In Sub- task A, the test dataset belongs to the same domain as the training and development datasets. These datasets are extracted from the suggestion forum for windows platform. In Subtask B, training and test datasets belong to separate domains. More specifically, the domain of the training dataset is entries from the windows forum and that of the test dataset is hotel reviews. Example sentences used in these tasks are listed in Table 1. For a de- scription of these tasks please refer to (Negi et al., 2019). Subtask A can be viewed as a binary text classi- fication task. Recently, pre-training models, such as OpenAI GPI (Radford et al., 2018) and BERT 1https://competitions.codalab.org/competitions/19955 2https://competitions.codalab.org/competitions/19956 Subtask A (windows platform) P: xbox dev mode companion work ipv6 please... N: I do not want to convert MSIs. Subtask B (hotel review) P: If you can, upgrade to an Ocean Front Room. N: it doesn’t have a very clean look. Table 1: Example sentences in the test dataset. P means a positive sentence, i.e. suggestion sentence and N de- notes a negative sentence, i.e. non-suggestion sentence. (Devlin et al., 2018), have gained much attention with their ability to improve a number of down- stream tasks. These models are pre-trained using unlabeled corpora and then fine-tuned on labeled datasets. We apply BERT to this task because it has achieved state-of-the-art performance in sev- eral text classification tasks. The difference to the original BERT model is that we further pre-train BERT model using an unlabeled corpus related to this domain. More concretely, we extract docu- ments from a windows forum and run additional steps of pre-training using these documents, start- ing from the pre-trained BERT model. In Subtask B, we apply the idea of distant supervision which has been firstly proposed by (Mintz et al., 2009). Distant …

Content truncated to 500 words