SemEval-2019 Task 12: Toponym Resolution in Scientific Papers

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 907–916 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 907 SemEval-2019 Task 12: Toponym Resolution in Scientific Papers Davy Weissenbacher†, Arjun Magge‡, Karen O’Connor†, Matthew Scotch‡, Graciela Gonzalez-Hernandez† †DBEI, The Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA ‡Biodesign Center for Environmental Health Engineering, Arizona State University, Tempe, AZ 85281, USA †{dweissen, karoc, gragon}@pennmedicine.upenn.edu ‡{amaggera, Matthew.Scotch}@asu.edu Abstract We present the SemEval-2019 Task 12 which focuses on toponym resolution in scientific ar- ticles. Given an article from PubMed, the task consists of detecting mentions of names of places, or toponyms, and mapping the mentions to their corresponding entries in GeoNames.org, a database of geospatial loca- tions. We proposed three subtasks. In Sub- task 1, we asked participants to detect all to- ponyms in an article. In Subtask 2, given to- ponym mentions as input, we asked partici- pants to disambiguate them by linking them to entries in GeoNames. In Subtask 3, we asked participants to perform both the de- tection and the disambiguation steps for all toponyms. A total of 29 teams registered, and 8 teams submitted a system run. We summarize the corpus and the tools created for the challenge. They are freely available at https://competitions.codalab. org/competitions/19948. We also an- alyze the methods, the results and the errors made by the competing systems with a focus on toponym disambiguation. 1 Introduction Toponym resolution, also known as geoparsing, geo-grounding or place name resolution, aims to assign geographic coordinates to all location names mentioned in documents. Toponym res- olution is usually performed in two independent steps. First, toponym detection or geotagging, where the span of place names mentioned in a doc- ument is noted. Second, toponym disambiguation or geocoding, where each name found is mapped to latitude and longitude coordinates correspond- ing to the centroid of its physical location. To- ponym detection has been extensively studied in named entity recognition: location names were one of the first classes of named entities to be detected in text (Piskorski and Yangarber, 2013). Disambiguation of toponyms is a more recent task (Leidner, 2007). With the growth of the internet, the public adop- tion of smartphones equipped with Geographic In- formation Systems and the collaborative devel- opment of comprehensive maps and geographi- cal databases, toponym resolution has seen an im- portant gain of interest in the last two decades. Not only academic but also commercial and open source toponym resolvers are now available. How- ever, their performance varies greatly when ap- plied on corpora of different genres and domains (Gritta et al., 2018). Toponym disambiguation tackles ambiguities between different toponyms, like Manchester, NH, USA vs. Manchester, UK (Geo-Geo ambiguities), and between toponyms and other entities, such as names of people or daily life objects (Geo-NonGeo ambiguities). Ad- ditional linguistic challenges during the resolution step may be metonymic usage of toponyms, “91% of the US didn’t vote for either Hilary or Trump” (a country does not vote, thus the toponym …

Content truncated to 500 words