Fair Evaluation in Concept Normalization: a Large-scale Comparative Analysis for BERT-based Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 662–668 Barcelona, Spain (Online), December 8-13, 2020 662 TIMBERT: Toponym Identifier For The Medical Domain Based on BERT MohammadReza Davari and Leila Kosseim and Tien D. Bui Dept. Computer Science and Software Engineering Concordia University, Montreal QC H3G 1M8, Canada mohammadreza.davari@mail.concordia.ca {leila.kosseim,tien.bui}@concordia.ca Abstract In this paper, we propose an approach to automate the process of place name detection in the medical domain to enable epidemiologists to better study and model the spread of viruses. We created a family of Toponym Identification Models based on BERT (TIMBERT), in order to learn in an end-to-end fashion the mapping from an input sentence to the associated sentence la- beled with toponyms. When evaluated with the SemEval 2019 task 12 test set (Weissenbacher et al., 2019), our best TIMBERT model achieves an F1 score of 90.85%, a significant improvement compared to the state-of-the-art of 89.13% (Wang et al., 2019). 1 Introduction Phylogeographers, who study the geographic distribution of viruses, have long linked the increase in the geographical spread of viruses (Gautret et al., 2012; Green and Roberts, 2000) to the growth in global tourism and international trade of goods. Most notably, in December 2019, a pneumonia-like disease, later dubbed COVID-19, that was detected in the city of Wuhan, China quickly became a world pandemic mainly due to global travels (World Health Organization, 2020b; World Health Organization, 2020a). Epidemiologists study the global impact of the spread of viruses by considering information on the DNA sequence and structure of viruses, as well as relying on accurate metadata. Although accurate localized geographical data is critical for creating maps of the locations of viruses and their migration paths, most publicly available databases, such as GenBank (Benson et al., 2012), provide insufficient details on the matter, limited to country or state level. In a study by Scotch et al. (2011), it is estimated that 7% of the GenBank records are missing geospatial metadata and 73% lacking detailed records. However, more fine-grained localization information on the viruses may be present in articles that describe the research work. Therefore, a manual inspection of biomedical articles is vital for obtaining more detailed information about the locations of the viruses. Toponym detection aims to identify the word boundaries of expressions that denote geographic names. Toponym detection has been the focus of much work in recent years (Ardanuy and Sporleder, 2017; De- Lozier et al., 2015; Taylor, 2017) and studies have shown that the task is highly dependent on the textual domain (Amitay et al., 2004; Purves et al., 2007; Qin et al., 2010; Kienreich et al., 2006; Garbin and Mani, 2005). The focus of this paper is to propose a competitive deep learning based model for toponym detection that learns in an end-to-end fashion the mapping from an input sentence to the associated sentence with toponym labels. We evaluated our models using the SemEval 2019 task 12 test set (Weissenbacher et al., 2019) and report their performances based on precision, recall and F1 …

Content truncated to 500 words