CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7888 - 7915 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark Ningyu Zhang1∗, Mosha Chen2∗, Zhen Bi1∗, Xiaozhuan Liang1∗, Lei Li1∗, Xin Shang3 Kangping Yin2, Chuanqi Tan2, Jian Xu2, Fei Huang2, Luo Si2, Yuan Ni4, Guotong Xie4,5,6 Zhifang Sui7,13, Baobao Chang7,13, Hui Zong8,14, Zheng Yuan9, Linfeng Li10, Jun Yan10 Hongying Zan11,13, Kunli Zhang11,13, Buzhou Tang12,13†, Qingcai Chen12,13† 1AZFT Joint Lab for Knowledge Engine, Zhejiang University 2Alibaba Group, 3School of Mathematical Science, Zhejiang University, 4Pingan Health Technology, 5Ping An Health Cloud Company Limited 6Ping An International Smart City Technology Co., Ltd 7Key Laboratory of Computational Linguistics, Ministry of Education, Peking University 8School of Life Sciences and Technology, Tongji University, 9Tsinghua University, 10Yidu Cloud Technology Inc 11School of Information Engineering, Zhengzhou University 12Harbin Institute of Technology (Shenzhen) 13Peng Cheng Laboratory, 14Philips Research China Abstract With the development of biomedical language understanding benchmarks, Artificial Intel- ligence applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it chal- lenging to replicate many of the successes in English for other languages. To facili- tate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Under- standing Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state- of-the-art neural models perform far worse than the human ceiling1. Our benchmark is released at https://tianchi.aliyun. com/dataset/dataDetail?dataId= 95414&lang=en-us. 1 Introduction Artificial intelligence is gradually changing the landscape of healthcare, and biomedical research (Yu et al., 2018). With the fast advancement of biomedical datasets, biomedical natural language processing (BioNLP) has facilitated a broad range ∗Equal contribution and shared co-first authorship. †Corresponding author. 1Code available in https://github.com/ CBLUEbenchmark/CBLUE of applications such as biomedical text mining, which leverages textual data in Electronic Health Records (EHRs). A key driving force behind such improvements and rapid iterations of models is the use of general evaluation datasets and benchmarks (Gijsbers et al., 2019). Pioneer benchmarks, such as BLURB (Gu et al., 2020), PubMedQA (Jin et al., 2019), and others, have provided us with the opportunity to conduct research on biomedical language under- standing and developing real-world applications. Unfortunately, most of these benchmarks are devel- oped in English, which makes the development of the associated machine intelligence Anglo-centric. Meanwhile, other languages, such as Chinese, have unique linguistic characteristics and categories that need to be considered. Even though Chinese speak- ers account for a quarter of the world population, there have been no existing Chinese biomedical language understanding evaluation benchmarks. To address this issue and facilitate natural lan- guage processing studies in Chinese, we take the first step in introducing a comprehensive Chinese Biomedical Language …

Content truncated to 500 words