Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 962–966 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 962 Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector Mehdi Drissi, Pedro Sandoval, Vivaswat Ojha, and Julie Medero Harvey Mudd College, CA {mdrissi, psandovalsegura, vmojha, jmedero}@hmc.edu Abstract We investigate the recently developed Bidirec- tional Encoder Representations from Trans- formers (BERT) model (Devlin et al., 2018) for the hyperpartisan news detection task. Us- ing a subset of hand-labeled articles from Se- mEval as a validation set, we test the perfor- mance of different parameters for BERT mod- els. We find that accuracy from two different BERT models using different proportions of the articles is consistently high, with our best- performing model on the validation set achiev- ing 85% accuracy and the best-performing model on the test set achieving 77%. We fur- ther determined that our model exhibits strong consistency, labeling independent slices of the same article identically. Finally, we find that randomizing the order of word pieces dramat- ically reduces validation accuracy (to approxi- mately 60%), but that shuffling groups of four or more word pieces maintains an accuracy of about 80%, indicating the model mainly gains value from local context. 1 Introduction SemEval Task 4 (Kiesel et al., 2019) tasked par- ticipating teams with identifying news articles that are misleading to their readers, a phenomenon of- ten associated with “fake news” distributed by par- tisan sources (Potthast et al., 2017). We approach the problem through transfer learning to fine-tune a model for the document classification task. We use the BERT model based on the implementation of the github repository pytorch-pretrained-bert1 on some of the data pro- vided by Task 4 of SemEval. BERT has been used to learn useful representations for a variety of nat- ural language tasks, achieving state of the art per- formance in these tasks after being fine-tuned (De- vlin et al., 2018). It is a language representation 1https://github.com/huggingface/pytorch-pretrained- BERT model that is designed to pre-train deep bidirec- tional representations by jointly conditioning on both left and right context in all layers. Thus, it may be able to adequately account for complex characteristics as such blind, prejudiced reason- ing and extreme bias that are important to reliably identifying hyperpartisanship in articles. We show that BERT performs well on hyper- partisan sentiment classification. We use unsuper- vised learning on the set of 600,000 source-labeled articles provided as part of the task, then train us- ing supervised learning for the 645 hand-labeled articles. We believe that learning on source- labeled articles would bias our model to learn the partisanship of a source, instead of the article. Ad- ditionally, the accuracy of the model on validation data labeled by article differs heavily when the ar- ticles are labeled by publisher. Thus, we decided to use a small subset of the hand-labeled articles as our validation set for all of our experiments. As the articles are too large for the model to be …

Content truncated to 500 words