Exploring Transformer Text Generation for Medical Dataset Augmentation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4699–4708 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4699 Exploring Transformer Text Generation for Medical Dataset Augmentation Ali Amin-Nejad1, Julia Ive1, Sumithra Velupillai2 1 Imperial College London, 2 King’s College London {ali.amin-nejad18, j.ive}@imperial.ac.uk, sumithra.velupillai@kcl.ac.uk Abstract Natural Language Processing (NLP) can help unlock the vast troves of unstructured data in clinical text and thus improve healthcare research. However, a big barrier to developments in this field is data access due to patient confidentiality which prohibits the sharing of this data, resulting in small, fragmented and sequestered openly available datasets. Since NLP model development requires large quantities of data, we aim to help side-step this roadblock by exploring the usage of Natural Language Generation in augmenting datasets such that they can be used for NLP model development on downstream clinically relevant tasks. We propose a methodology guiding the generation with structured patient information in a sequence-to-sequence manner. We experiment with state-of-the-art Transformer models and demonstrate that our augmented dataset is capable of beating our baselines on a downstream classification task. Finally, we also create a user interface and release the scripts to train generation models to stimulate further research in this area. Keywords: Natural Language Generation, Language Modelling, Ethics and Legal Issues 1. Introduction Natural Language Processing (NLP) has enormous poten- tial to advance many aspects of healthcare by facilitating the analysis of unstructured text (Esteva et al., 2019). How- ever a key obstacle to the development of more powerful NLP methods in the clinical domain is a lack of accessi- ble data. This, coupled with the fact that state-of-the-art (SOTA) neural models are well known to require very large volumes of data in order to learn general and meaningful patterns, means that progress is hindered in this area. Data access is usually restricted due to the constraints on sharing personal medical information for confidentiality reasons, be they legal or ethical in nature (Chapman et al., 2011). In the Machine Learning community, similar problems are typically solved by using artificially generated data to aug- ment or perhaps even replace an original dataset (Bach- man, 2016) in e.g. image processing. However, similar approaches to data augmentation are not easily applied to NLP. With language being inherently more complex than other domains, it is difficult to programmatically modify a sentence or document without altering the meaning and coherency. Natural Language Generation (NLG) can pro- vide a more sophisticated approach to solving this problem and has already done so, e.g. in machine-translation with the technique known as back-translation (Sennrich et al., 2016). With newer, more capable, NLG models - utilis- ing the Transformer architecture (Vaswani et al., 2017) - we posit that this general idea can now be extended beyond machine translation to longer passages of text. Indeed NLG is an active area of NLP research, however there are still challenges to be addressed. The replace- ment or augmentation of genuine training data with arti- ficial …

Content truncated to 500 words