Annotating Spin in Biomedical Scientific Publications : the case of Random Controlled Trials (RCTs)

Annotating Spin in Biomedical Scientific Publications: the case of Randomized Controlled Trials (RCTs) Anna Koroleva, Patrick Paroubek LIMSI-CNRS, Bât. 508, rue John von Neumann, campus Université d'Orsay, F-91405 Orsay {koroleva, pap}@limsi.fr Abstract In this paper we report on the collection in the context of the MIROR project of a corpus of biomedical articles for the task of automatic detection of inadequate claims (spin), which to our knowledge has never been addressed before. We present the manual annotation model and its annotation guidelines and describe the planned machine learning experiments and evaluations. Keywords: spin, annotation scheme, biomedical articles 1. Introduction Merriam Webster dictionary defines spin doctor as “a person (such as a political aide) responsible for ensuring that others interpret an event from a particular point of view"1. In the context of the MIROR2 project, we address spin in biomedical scientific publications, where it refers to misleading presentation of scientific results, in particular in articles reporting randomized controlled trials (RCTs), an important type of clinical trial. In our case, spin consists in presenting the examined treatment as having greater beneficial effects than the experiments show (Boutron et al. 2010; Boutron et al. 2014; Haneef et al. 2015; Yavchitz et al. 2016 ). Spin in RCTs affects clinical decision-making (Boutron et al. 2014) and results in distorted presentation of research findings in media (Yamamoto & Takagi 2005; Hall et al. 2009). We present here the first steps aiming at proposing an algorithm for automatic spin identification in biomedical abstracts, something which to the best of our knowledge has not been attempted before. We present here the construction and annotation of a corpus of medical publication extracted from PubMed Central3 (PMC) about RCT and describe the annotation model and guidelines. 1.1 On spin types From previous research on spin classification (Boutron et al. 2010; Lazarus et al. 2015, Yavchitz et al. 2012], we can outline three main types and their subtypes of spin in RCT reports: 1. misleading reporting of study results: • selective reporting of outcomes (omission of the primary outcome; focus on statistically significant results different from the main outcome); • occulting adverse events; • misleading reporting of study design; • linguistic spin (beautifying formulations); • discarding limitations; 1https://www.merriam-webster.com/dictionary 2http://miror-ejd.eu/ 3https://www.ncbi.nlm.nih.gov/pmc/ • selective citation of other studies 2. inadequate interpretation of the results: • claiming a beneficial effect of the intervention despite statistically non-significant results; • claiming an equivalent effect of the interventions for statistically non-significant results; • claiming that the treatment is safe for statistically non-significant safety outcomes; • concluding a beneficial effect despite no comparison test performed; • interpretation of the results according to statistical significance instead of clinical relevance; 3. inadequate extrapolation: • inadequate extrapolation from the population, interventions or outcome actually assessed in the study to a larger population, different interventions or outcomes; • inadequate implications for clinical practice. Example of spin putting focus on secondary result (“improved PFS and response for treatment”) instead of the main result, object of the experiment (“survival rate”): “This study demonstrates …

Content truncated to 500 words