Creating a Dataset for Named Entity Recognition in the Archaeology Domain

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4573–4577 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4573 Creating a Dataset for Named Entity Recognition in the Archaeology Domain Alex Brandsen, Suzan Verberne, Milco Wansleeben, Karsten Lambers Leiden University Einsteinweg 2, 2333CC, Leiden, The Netherlands {a.brandsen, m.wansleeben, k.lambers}@arch.leidenuniv.nl s.verberne@liacs.leidenuniv.nl Abstract In this paper, we present the development of a training dataset for Dutch Named Entity Recognition (NER) in the archaeology domain. This dataset was created as there is a dire need for semantic search within archaeology, in order to allow archaeologists to find structured information in collections of Dutch excavation reports, currently totalling around 60,000 (658 million words) and growing rapidly. To guide this search task, NER is needed. We created rigorous annotation guidelines in an iterative process, then instructed five archaeology students to annotate a number of documents. The resulting dataset contains ~31k annotations between six entity types (artefact, time period, place, context, species & material). The inter-annotator agreement is 0.95, and when we used this data for machine learning, we observed an increase in F1 score from 0.51 to 0.70 in comparison to a machine learning model trained on a dataset created in prior work. This indicates that the data is of high quality, and can confidently be used to train NER classifiers. Keywords: Corpus Creation, Named Entity Recognition, Text Mining 1. Introduction The archaeology domain, like other scientific fields, pro- duces large amounts of textual data. Specifically, a large amount of excavation reports are available, which are cre- ated whenever an excavation is completed, detailing every- thing that has been found together with an interpretation of the site (Richards et al., 2015). In the Netherlands, this corpus is estimated at 70,000 documents, and is growing by 4000 each year (Rijksdienst voor het Cultureel Erfgoed, 2019). Most of these reports are created and published by individual commercial archaeology companies after they excavate, in low numbers and not widely shared. This so-called grey literature is currently underused, even though most scholars agree that the information hidden in these reports is of immense value (Evans, 2015). The sys- tems currently available to explore this corpus are metadata search engines that simply do not offer enough granularity for archaeologists to easily find what they are looking for. An example might be a single find from the Bronze Age which was not included in the temporal metadata as it is too specific. Currently, there is no way of finding this so called ‘by-catch’; single finds of a different type than the rest of the excavation. Users of the currently available search en- gines report they download whole portions of the available data and manually search through PDF files one by one to find the information they are looking for (Brandsen et al., 2019). Free text search across the entire corpus would already be a vast improvement, however this does not account for pol- ysemy and synonymy, which occur often in archaeological texts. …

Content truncated to 500 words