Transformers and the Representation of Biomedical Background Knowledge

Transformers and the Representation of Biomedical Background Knowledge Oskar Wysocki∗ Digital Experimental Cancer Medicine Team, Cancer Biomarker Centre CRUK Manchester Institute University of Manchester oskar.wysocki@manchester.ac.uk Zili Zhou Department of Computer Science University of Manchester zili.zhou@manchester.ac.uk Paul O’Regan Digital Experimental Cancer Medicine Team, Cancer Biomarker Centre CRUK Manchester Institute University of Manchester paul.oregan@digitalecmt.com Deborah Ferreira Department of Computer Science University of Manchester deborah.ferreira@manchester.ac.uk Magdalena Wysocka Digital Experimental Cancer Medicine Team, Cancer Biomarker Centre CRUK Manchester Institute University of Manchester magdalena.wysocka@digitalecmt.org ∗ Kilburn Building, Oxford Rd, Manchester M13 9PL, United Kingdom. E-mail: oskar.wysocki@manchester.ac.uk. Secondary affiliation: Department of Computer Science, University of Manchester. † Other affiliations: Digital Experimental Cancer Medicine Team, Cancer Biomarker Centre, CRUK Manchester Institute, University of Manchester; Department of Computer Science, University of Manchester. Action Editor: Byron Wallace. Submission received: 16 March 2022; revised version received: 18 August 2022; accepted for publication: 7 September 2022. https://doi.org/10.1162/coli a 00462 © 2022 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 49, Number 1 D´onal Landers Digital Experimental Cancer Medicine Team, Cancer Biomarker Centre CRUK Manchester Institute University of Manchester donal.landers@delondraoncology.com Andr´e Freitas† Idiap Research Institute Martigny, Switzerland andre.freitas@manchester.ac.uk Specialized transformers-based models (such as BioBERT and BioMegatron) are adapted for the biomedical domain based on publicly available biomedical corpora. As such, they have the poten- tial to encode large-scale biological knowledge. We investigate the encoding and representation of biological knowledge in these models, and its potential utility to support inference in cancer precision medicine—namely, the interpretation of the clinical significance of genomic alterations. We compare the performance of different transformer baselines; we use probing to determine the consistency of encodings for distinct entities; and we use clustering methods to compare and contrast the internal properties of the embeddings for genes, variants, drugs, and diseases. We show that these models do indeed encode biological knowledge, although some of this is lost in fine-tuning for specific tasks. Finally, we analyze how the models behave with regard to biases and imbalances in the dataset. 1. Introduction Transformers are deep learning models that are able to capture linguistic patterns at scale. By using unsupervised learning tasks that can be defined over large-scale textual corpora, these models are able to capture both linguistic and domain knowledge, which can be later specialized for specific inference tasks. The representation produced by the model is a high-dimensional linguistic space that represents words, terms, and sentences as vector projections. In Natural Language Processing, transformers are used to support natural language inference and classification tasks. The assumption is that the models can encode syntactic, semantic, commonsense, and domain-specific knowl- edge and use their internal representation for complex textual interpretation. While these models provided measurable improvements in many different tasks, the limited interpretability of their internal representation challenges their application in areas such as biomedicine. In this work we elucidate a set of the internal properties of transformers in the context of a well-defined cancer precision medicine inference task, in which the domain …

Content truncated to 500 words