ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1254–1261 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1254 ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples Cheoneum Park*1 and Juae Kim*2 and Hyeon-gu Lee*1 and Reinald Kim Amplayo*3 Harksoo Kim1 and Jungyun Seo2 and Changki Lee1 (* equal contribution) 1Kangwon National University, South Korea 2Sogang University, South Korea 3University of Edinburgh, UK Abstract This paper describes our system, Joint Encoders for Stable Suggestion Inference (JESSI), for the SemEval 2019 Task 9: Sug- gestion Mining from Online Reviews and Fo- rums. JESSI is a combination of two sentence encoders: (a) one using multiple pre-trained word embeddings learned from log-bilinear re- gression (GloVe) and translation (CoVe) mod- els, and (b) one on top of word encodings from a pre-trained deep bidirectional trans- former (BERT). We include a domain adver- sarial training module when training for out- of-domain samples. Our experiments show that while BERT performs exceptionally well for in-domain samples, several runs of the model show that it is unstable for out-of- domain samples. The problem is mitigated tremendously by (1) combining BERT with a non-BERT encoder, and (2) using an RNN- based classifier on top of BERT. Our final models obtained second place with 77.78% F-Score on Subtask A (i.e. in-domain) and achieved an F-Score of 79.59% on Subtask B (i.e. out-of-domain), even without using any additional external data. 1 Introduction Opinion mining (Pang and Lee, 2007) is a huge field that covers many NLP tasks ranging from sentiment analysis (Liu, 2012), aspect extraction (Mukherjee and Liu, 2012), and opinion summa- rization (Ku et al., 2006), among others. De- spite the vast literature on opinion mining, the task on suggestion mining has given little attention. Suggestion mining (Brun and Hag`ege, 2013) is the task of collecting and categorizing suggestions about a certain product. This is important because while opinions indirectly give hints on how to im- prove a product (e.g. analyzing reviews), sugges- tions are direct improvement requests (e.g. tips, advice, recommendations) from people who have used the product. To this end, Negi et al. (2019) organized a shared task specifically on suggestion mining called SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. The shared task is composed of two subtasks, Subtask A and B. In Subtask A, systems are tasked to predict whether a sentence of a certain domain (i.e. electronics) entails a suggestion or not given a training data of the same domain. In Subtask B, systems are tasked to do suggestion prediction of a sentence from another domain (i.e. hotels). Organizers observed four main challenges: (a) sparse occur- rences of suggestions; (b) figurative expressions; (c) different domains; and (d) complex sentences. While previous attempts (Ramanand et al., 2010; Brun and Hag`ege, 2013; Negi and Buitelaar, 2015) made use of human-engineered features to solve this problem, the goal of the shared task is to lever- age the advancements seen on neural networks, by providing a larger dataset …

Content truncated to 500 words