Exploring Large Language Models for Classical Philology

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15181–15199 July 9-14, 2023 ©2023 Association for Computational Linguistics Exploring Large Language Models for Classical Philology Frederick Riemenschneider Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg riemenschneider@cl.uni-heidelberg.de Anette Frank Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg frank@cl.uni-heidelberg.de Abstract Recent advances in NLP have led to the cre- ation of powerful language models for many languages including Ancient Greek and Latin. While prior work on Classical languages unani- mously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages: we explore (i) encoder-only and encoder-decoder architectures using ROBERTA and T5 as strong model types, and create for each of them (ii) a monolingual Ancient Greek and a multi- lingual instance that includes Latin and English. We evaluate all models on morphological and syntactic tasks, including lemmatization, which demonstrates the added value of T5’s decod- ing abilities. We further define two probing tasks to investigate the knowledge acquired by models pre-trained on Classical texts. Our ex- periments provide the first benchmarking ana- lysis of existing models of Ancient Greek. Re- sults show that our models provide significant improvements over the SoTA. The systematic analysis of model types can inform future re- search in designing language models for Clas- sical languages, including the development of novel generative tasks. We make all our mod- els available as community resources, along with a large curated pre-training corpus for An- cient Greek, to support the creation of a larger, comparable model zoo for Classical Philol- atogy. Our models and resources are available https://github.com/Heidelberg-NLP/ ancient-language-models. 1 Introduction Since the beginning of the creation of the Index Thomisticus in 1946 (Busa, 1980) and the publica- tion of the Concordance to Livy (Packard, 1968), Classical Philology has been revitalized by the “dig- ital revolution” (Berti, 2019). Today, numerous ef- forts have been undertaken to make Classical texts digitally available, annotate, and automatically pro- cess them. E.g., the Classical Language Toolkit (CLTK, Johnson et al., 2021) offers various tools to process pre-modern languages, in particular Latin and pre-modern Greek.1 Recently, we see a surge of the first pre-trained contextualized language models (PLMs) for Classi- cal languages: Latin BERT has been proposed by Bamman and Burns (2020), Ancient Greek (AG) BERT by Singh et al. (2021). Lately, a second AG BERT has been proposed by Yamshchikov et al. (2022). However, both AG BERT models have been pre-trained on a comparatively small pre- training dataset. Moreover, they have been initial- ized from Modern Greek BERT (Koutsikakis et al., 2020), which limits them to the modern Greek al- phabet, ignoring the diacritics of Ancient Greek. Although numerous richly annotated treebanks are available for Latin and AG, systems have, by now, not been evaluated on a shared benchmark. Given that two popular treebanks for AG have been integrated into Universal Dependencies (de Marn- effe et al., 2021), …

Content truncated to 500 words