SemEval-2019 Task 4: Hyperpartisan News Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 829–839 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 829 SemEval-2019 Task 4: Hyperpartisan News Detection Johannes Kiesel1 Maria Mestre2,3 Rishabh Shukla2 Emmanuel Vincent2 Payam Adineh1 David Corney4 Benno Stein1 Martin Potthast5 1 Bauhaus-Universität Weimar <first>.<last>@uni-weimar.de 2 Factmata Ltd. <first>.<last>@factmata.com 3 mariarmestre@gmail.com 4 dpacorney@gmail.com 5 Leipzig University martin.potthast@uni-leipzig.de Abstract Hyperpartisan news is news that takes an ex- treme left-wing or right-wing standpoint. If one is able to reliably compute this meta in- formation, news articles may be automatically tagged, this way encouraging or discouraging readers to consume the text. It is an open ques- tion how successfully hyperpartisan news de- tection can be automated, and the goal of this SemEval task was to shed light on the state of the art. We developed new resources for this purpose, including a manually labeled dataset with 1,273 articles, and a second dataset with 754,000 articles, labeled via distant supervi- sion. The interest of the research community in our task exceeded all our expectations: The datasets were downloaded about 1,000 times, 322 teams registered, of which 184 configured a virtual machine on our shared task cloud ser- vice TIRA, of which in turn 42 teams sub- mitted a valid run. The best team achieved an accuracy of 0.822 on a balanced sample (yes : no hyperpartisan) drawn from the manu- ally tagged corpus; an ensemble of the submit- ted systems increased the accuracy by 0.048. 1 Introduction Yellow journalism has established itself in so- cial media, nowadays often linked to phenomena like clickbait, fake news, and hyperpartisan news. Clickbait has been its first “success story” (Potthast et al., 2016): When the viral spreading of pieces of information was first observed in social networks, some investigated how to manufacture such events for profit. Unlike for “natural” viral content, how- ever, readers had to be directed to a web page containing the to-be-spread information alongside paid-for advertising, so that only teasers and not the information itself could be shared. Then, to maximize their virality, data-driven optimization revealed that teaser messages which induce curios- ity, or any other kind of strong emotion, spread best. The many forms of such teasers that have emerged since are collectively called clickbait. New pub- lishing houses arose around viral content, which brought clickbait into the mainstream. Traditional news publishers, struggling for their share of the attention market that is a social network, adopted clickbait into their toolbox, too, despite its violation of journalistic codes of ethics. The content spread using clickbait used to be mostly harmless trivia—entertainment and distrac- tion to some, spam to others—, but in the wake of the 2016 United States presidential election, “fake news” came to widespread public attention. While certainly not a new phenomenon in yellow journal- ism, its viral success on social media was a surprise to many. Part of this success was then attributed to so-called hyperpartisan news publishers (Bhatt et al., 2018), which report strongly in favor of …

Content truncated to 500 words