TMLab SRPOL at SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1172–1175 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1172 TMLab SRPOL at SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums Piotr Niewi´nski, Aleksander Wawer, Maria Pszona, Maria Janicka Samsung R&D Institute Poland pl. Europejski 1 00-844 Warsaw, Poland {p.niewinski, a.wawer, m.pszona, m.janicka}@samsung.com Abstract The article describes our submission to SemE- val 2019 Task 8 on Fact-Checking in Com- munity Forums. The systems under discus- sion participated in Subtask A: decide whe- ther a question asks for factual information, opinion/advice or is just socializing. Our pri- mary submission was ranked as the second one among all participants in the official evaluation phase. The article presents our primary solu- tion: Deeply Regularized Residual Neural Ne- twork (DRR NN) with Universal Sentence En- coder embeddings. This is followed by a de- scription of two contrastive solutions based on ensemble methods. 1 Introduction Community question answering forums are good platforms for knowledge sharing; hence, they are widely used sources of information. The growing popularity of such knowledge exchange leads to a growing need to automate the process of verifying the post quality. The first step, often overlooked, is to categorize each question and establish what kind of information the user seeks. Question classification has been mainly used to support question answering systems. Two main method types have been proposed in the litera- ture: (1) rule-based approaches with linguistic fe- atures (Tomuro, 2004; Huang et al., 2008; Silva et al., 2011), and (2) machine learning approaches (Zhang and Lee, 2003; Metzler and Croft, 2005). These methods are rather simple, due to the fact that question classification is often just a prepro- cessing step in a larger task. However, we can ob- serve some recent advances in this area, such as ULMFiT (Howard and Ruder, 2018), which achie- ves state-of-the-art performance on the TREC da- taset (Voorhees and Tice, 1999). The present article describes our systems sub- mitted to the SemEval 2019 competition Task 8 subtask A on question classification. The com- petition data set consisted of QatarLiving forum questions classified as FACTUAL, OPINION or SOCIALIZING. The training data contained only 1,118 questions. Moreover, according to our eva- luation, human-level accuracy on this data set was about 0.75, which was relatively low. Therefore, we approached the task as a challenging classifi- cation problem. The article is structured as follows. Section 2 presents our experiments with preprocessing me- thods. Section 3 describes our official submission, where we propose an architecture utilizing several regularization methods to address the problem of the small data set. For comparative purposes, sec- tion 4 presents two ensemble models as contrastive examples. Section 5 provides the results achieved by the models. Lastly, section 6 concludes the di- scussion. 2 Data Preprocessing We tested a few simple text preprocessing setups. Unfortunately, none of them helped the models achieve improved results. Hence, they are here presented as negative results, and for reference. First, all emojis were removed from …

Content truncated to 500 words