Relation Extraction Using Multiple Pre-Training Models in Biomedical Domain

Proceedings of Recent Advances in Natural Language Processing, pages 530–537 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_060 530 Relation Extraction using Multiple Pre-Training Models in Biomedical Domain Satoshi Hiai1, Kazutaka Shimada1, Taiki Watanabe2, Akiva Miura2∗, Tomoya Iwakura2 1Kyushu Institute of Technology, Fukuoka, Japan 2Fujitsu Ltd., Tokyo, Japan 3Atid Ltd., Tokyo, Japan s hiai@pluto.ai.kyutech.ac.jp, shimada@ai.kyutech.ac.jp {watanabe-taiki, iwakura.tomoya}@fujitsu.com akiva.miura@studio-atid.com Abstract The number of biomedical documents is in- creasing rapidly. Accordingly, a demand for extracting knowledge from large-scale biomedi- cal texts is also increasing. BERT-based models are known for their high performance in vari- ous tasks. However, it is often computationally expensive. A high-end GPU environment is not available in many situations. To attain both high accuracy and fast extraction speed, we propose combinations of simpler pre-trained models. Our method outperforms the latest state-of-the-art model and BERT-based models on the GAD corpus. In addition, our method shows approximately three times faster extrac- tion speed than the BERT-based models on the ChemProt corpus and reduces the memory size to one sixth of the BERT ones. 1 Introduction The amount of biomedical documents is increasing rapidly. The documents contain valuable knowl- edge, such as chemical compound names and their relations. However, the current knowledge extrac- tion is considerably manual. Therefore, a demand for extracting knowledge automatically from large- scale biomedical text data is increasing. Biomedical relation extraction (RE) models based on BERT (Devlin et al., 2019) have shown great performance (Lee et al., 2019; Beltagy et al., 2019). The methods using BERT models pre- trained on biomedical corpora achieved state-of- the-art (SOTA) performance on several biomedical RE datasets. However, BERT models require a huge amount of computational resources and gen- erally need a long time for extraction processes. By processing data in parallel with multiple computa- tional resources, we can process larger text as com- pared with a single resource. However, a high-end ∗Work done while the author was at Fujitsu Laboratories Ltd. GPU or a distributed environment for efficient com- putation is not available in many situations. Even if we can utilize such computational resources, sub- stantial energy consumption becomes a problem (Strubell et al., 2019). Therefore, more lightweight and accurate RE models are expected. In this paper, we construct a biomedical RE model by combining word embeddings obtained from multiple lightweight models. The RE model can be executed in a wide range of environments, such as a CPU and a middle-class GPU, by re- ducing memory consumption during the learning process or the inference process. During the infer- ence process of our proposed model, the amount of calculation can be suppressed and the model can process documents at high speed. Since the cal- culation in each lightweight model is small scale, memory consumption can be suppressed. Further- more, by selecting whether or not to utilize each word embeddings, we can customize the model to be suitable for the computer environment. Hence, our model can process faster than the BERT-based models in the inference process. We adopt more lightweight pre-trained models: ELMo (Peters et al., 2018) and …

Content truncated to 500 words