MedICaT: A Dataset of Medical Images, Captions, and Textual References

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2112–2120 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2112 MedICaT: A Dataset of Medical Images, Captions, and Textual References Sanjay Subramanian1 Lucy Lu Wang1 Sachin Mehta2 Ben Bogin3 Madeleine van Zuylen1 Sravanthi Parasa4 Sameer Singh5 Matt Gardner1 Hannaneh Hajishirzi1,2 1 Allen Institute for AI 2 University of Washington 3 Tel Aviv University 4 Swedish Medical Group 5 University of California, Irvine {sanjays, lucyw}@allenai.org Abstract Understanding the relationship between fig- ures and text is key to scientific document un- derstanding. Medical figures in particular are quite complex, often consisting of several sub- figures (75% of figures in our dataset), with de- tailed text describing their content. Previous work studying figures in scientific papers fo- cused on classifying figure content rather than understanding how images relate to the text. To address challenges in figure retrieval and figure-to-text alignment, we introduce MED- ICAT, a dataset of medical images in con- text. MEDICAT consists of 217K images from 131K open access biomedical papers, and includes captions, inline references for 74% of figures, and manually annotated sub- figures and subcaptions for a subset of fig- ures. Using MEDICAT, we introduce the task of subfigure to subcaption alignment in compound figures and demonstrate the util- ity of inline references in image-text match- ing. Our data and code can be accessed at https://github.com/allenai/medicat. 1 Introduction Scientific document understanding necessitates the analysis of various components of scientific papers, including recognizing relationships between the text, figures, and references of papers. In the medi- cal domain, connections between the text and fig- ures in a paper are useful to enable the retrieval of figures via textual queries and to produce systems that are capable of analyzing and understanding medical images. Modern academic search engines are able to ex- tract figures and associated captions from papers, so that queries can be matched against captions to retrieve relevant images. However, scientific fig- ures often contain subfigures (40% of figures in PubMed (De Herrera et al., 2016)), and for a given Figure 1: Example of color-coded subfigures and cor- responding subcaptions (top), with an example inline reference from the full text. Figure and text adapted from Dhungana et al. (2018).1 query, only some of these may be relevant. Con- sider a user searching “lung cyst CT”; ideally, the user would be shown just parts (b) and (d) from Figure 1. But selecting relevant subfigures requires finding and aligning subcaptions with subfigures, and current systems lack this ability. Existing large- scale datasets (Pelka et al., 2018; Ionescu et al., 2018) explicitly exclude compound figures, while datasets with subfigure annotations (De Herrera et al., 2016; You et al., 2011) do not provide an- notations for aligning subfigures with subcaptions, and therefore cannot support subfigure retrieval. Images and textual descriptions in medical pa- pers are also useful for developing systems for au- tomated medical image analysis. Previous work (Ionescu et al., 2018; Pelka et al., 2019) has used such data for tasks such …

Content truncated to 500 words