ColumbiaNLP at SemEval-2019 Task 8: The Answer is Language Model Fine-tuning

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1144–1148 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1144 ColumbiaNLP at SemEval-2019 Task 8: The Answer is Language Model Fine-tuning Tuhin Chakrabarty Columbia University Department Of Computer Science tc2896@columbia.edu Smaranda Muresan Columbia University Data Science Institute smara@columbia.edu Abstract Community Question Answering forums are very popular nowadays, as they represent ef- fective means for communities to share infor- mation around particular topics. But the in- formation shared on these forums is often not correct or misleading. This paper presents the ColumbiaNLP submission for the SemEval- 2019 Task 8: Fact-Checking in Community Question Answering Forums. We show how fine-tuning a language model on a large unan- notated corpus of old threads from Qatar Liv- ing forum helps us to classify question types (factual, opinion, socializing) and to judge the factuality of answers on the shared task labeled data from the same forum. Our system fin- ished 4th and 2nd on Subtask A (question type classification) and B (answer factuality predic- tion), respectively, based on the official metric of accuracy. 1 Introduction Community Question Answering (cQA) forums such as StackOverflow, Yahoo! Answers, and Quora are very popular nowadays, as they repre- sent effective means for communities to share in- formation and to collectively satisfy their informa- tion needs. Questions asked on these sites can be of different types, and the answers can often be false, misleading or irrelevant. SemEval-2019 Task 8 is structured around two subtasks. Subtask A is a question classification task, where the questions types are: • Factual: The question is asking for fac- tual information, which can be answered by checking various information sources, and it is not ambiguous (e.g., “What is Ooredoo customer service number?”). • Opinion: The question asks for an opinion or an advice, not for a fact. (e.g., “Can anyone recommend a good Vet in Doha?””) • Socializing: Not a real question, but intended for socializing or for chatting. This can also mean expressing an opinion or sharing some information, without really asking anything of general interest (e.g., “What was your first car?”) Subtask B is an answer classification task: are the answers to factual questions factual or not, and if they are factual are they true or false: • Factual - TRUE: The answer is True and can be proven with an external resource. (Q: “I wanted to know if there were any specific shots and vaccinations I should get before coming over [to Doha].”; A: “Yes there are; though it varies depending on which coun- try you come from. In the UK; the doctor has a list of all countries and the vaccinations needed for each.”). • Factual - FALSE: The answer gives a factual response, but it is False, it is partially false or the responder is unsure about (Q:“Can I bring my pitbulls to Qatar?”; A: “Yes you can bring it but be careful this kind of dog is very dangerous.”). • Non-Factual: When the answer does …

Content truncated to 500 words