Detecting Health Advice in Medical Research Literature

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6018–6029 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6018 Detecting Health Advice in Medical Research Literature Yingya Li School of Information Studies Syracuse University yli48@syr.edu Jun Wang Independent Researcher Syracuse, NY junwang4@gmail.com Bei Yu School of Information Studies Syracuse University byu@syr.edu Abstract Health and medical researchers often give clin- ical and policy recommendations to inform health practice and public health policy. How- ever, no current health information system sup- ports the direct retrieval of health advice. This study fills the gap by developing and validating an NLP-based prediction model for identifying health advice in research publications. We an- notated a corpus of 6,000 sentences extracted from structured abstracts in PubMed publica- tions as “strong advice”, “weak advice”, or “no advice”, and developed a BERT-based model that can predict, with a macro-averaged F1- score of 0.93, whether a sentence gives strong advice, weak advice, or not. The prediction model generalized well to sentences in both unstructured abstracts and discussion sections, where health advice normally appears. We also conducted a case study that applied this predic- tion model to retrieve specific health advice on COVID-19 treatments from LitCovid, a large COVID research literature portal, demonstrat- ing the usefulness of retrieving health advice sentences as an advanced research literature navigation function for health researchers and the general public. 1 Introduction Clinical practice and public health policies need to be guided by evidence presented in peer-reviewed medical literature, where researchers present their findings and discuss implications (Schaafsma et al., 2005). Sometimes, researchers would even give clinical and policy recommendations, defined as “health advice” in this study. However, whether to give health advice in medical research papers is a controversial issue. The opponents are con- cerned about the quality of health advice given in individual research papers in that a single pa- per lacks sufficient information for all evidence in real practice and there is limited manuscript space for full review of alternative choices (Cummings, 2007). For example, some medical experts warned that a large proportion of health advice inferred from over-interpreted observational results was not fully supported by the study presented (Wilson and Chestnutt, 2016; Banerjee and Prasad, 2020), although such health advice is frequently found in medical publications, even in highly influential journals (Prasad et al., 2013). On the other hand, proponents of “actionable research” would like to encourage more efficient and effective transmission of science evidence into practice (Green et al., 2009). Giving health advice may also benefit the general public. Some journal editors suggested “keeping the needs of less sci- entifically inclined readers in mind” (Pless, 2007). After all, if researchers themselves do not discuss the practical value of their findings, press officers and journalists might misinterpret the results and give exaggerated health advice in press releases and news articles (Sumner et al., 2014; Haneef et al., 2015). Both arguments for and against giving health advice in individual studies indicate a strong need for identifying and accessing …

Content truncated to 500 words