Transfer Learning for Czech Historical Named Entity Recognition

Proceedings of Recent Advances in Natural Language Processing, pages 576–582 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_065 576 Transfer learning for Czech Historical Named Entity Recognition Helena Hubkov´a†‡ † Department of Computer Science and Engineering, University of West Bohemia, Plzeˇn, Czech Republic hhubkova@kiv.zcu.cz Pavel Kr´al†‡ ‡ NTIS – New Technologies for the Information Society, University of West Bohemia, Plzeˇn, Czech Republic pkral@kiv.zcu.cz Abstract Nowadays, named entity recognition (NER) achieved excellent results on the standard cor- pora. However, big issues are emerging with a need for an application in a specific domain, because it requires a suitable annotated corpus with adapted NE tag-set. This is particularly evident in the historical document processing field. The main goal of this paper consists of propos- ing and evaluation of several transfer learning methods to increase the score of the Czech historical NER. We study several information sources, and we use two neural nets for NE modeling and recognition. We employ two corpora for evaluation of our transfer learning methods, namely Czech named entity corpus and Czech historical named entity corpus. We show that BERT rep- resentation with fine-tuning and only the sim- ple classifier trained on the union of corpora achieves excellent results. 1 INTRODUCTION Recently, named entity recognition (NER) achieved outstanding results on standard NER corpora. Par- ticularly on English CONLL-2003 corpus 90% F- measure has been overcome, which is sufficient for several real applications. However, big issues are emerging with a need for an application in the specific domain which re- quires an appropriately annotated corpus including adapted NE tag-set. This issue is particularly evi- dent in the case of historical documents in Czech language on which we focus on the project ”Mod- ern Access to Historical Sources”1. Manual an- notation of this corpus is very expensive and time consuming. Moreover, the presence of a linguist is necessary. We will use the information about 1http://www.portafontium.eu/ named entities as additional metadata for informa- tion retrieval and knowledge extraction. Transfer learning targets at reusing information obtained from one corpus to improve the results of a model learned on an analogous task with few resources. To overcome this issue, we propose and evaluate several transfer learning approaches to im- prove the results of the Czech historical NER when the annotated resources are limited. The following information sources are considered and studied for this task: • pre-trained fastText2 word vectors; • pre-trained word2vec3 word vectors; • pre-trained Slavic BERT4 contextual text rep- resentation; • Czech contemporary NE corpus from differ- ent domain. We employ two neural based models, namely recurrent Bidirectional long short-term memory (BiLSTM) (Graves et al., 2005) and Bidirectional encoder representations from transformers (BERT) model (Devlin et al., 2019) with a simple percep- tron for NER modelling and recognition. We use two corpora for evaluation of our experiments. Note that, to the best of our knowledge, this is the first attempt to employ transfer learning in the field of Czech historical NER. 2 RELATED WORK Rodriguez et al. (Rodriguez et al., 2018) presented reproduction paper focused on …

Content truncated to 500 words