Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4682–4691 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4682 Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach Koren Lazar♢ Benny Saret♠ Asaf Yehudai♢ Wayne Horowitz♠ Nathan Wasserman♠ Gabriel Stanovsky♢ ♢School of Computer Science and Engineering ♠The Institute of Archaeology The Hebrew University of Jerusalem, Jerusalem, Israel {koren.lazar, gabriel.stanovsky}@mail.huji.ac.il Abstract We present models which complete miss- ing text given transliterations of ancient Mesopotamian documents, originally written on cuneiform clay tablets (2500 BCE - 100 CE). Due to the tablets’ deterioration, schol- ars often rely on contextual cues to manually fill in missing parts in the text in a subjective and time-consuming process. We identify that this challenge can be formulated as a masked language modelling task, used mostly as a pre- training objective for contextualized language models. Following, we develop several archi- tectures focusing on the Akkadian language, the lingua franca of the time. We find that de- spite data scarcity (1M tokens) we can achieve state of the art performance on missing tokens prediction (89% hit@5) using a greedy decod- ing scheme and pretraining on data from other languages and different time periods. Finally, we conduct human evaluations showing the ap- plicability of our models in assisting experts to transcribe texts in extinct languages. 1 Introduction The Akkadian language was the lingua franca of the Middle East and Egypt in the Late Bronze and Early Iron Ages, spoken or in use from 2500 BCE until its gradual extinction around 100 CE (Oppen- heim, 2013). It was written in cuneiform signs — wedge-shaped imprints on clay tablets, as depicted in Figure 1 (Walker, 1987). These tablets are the main record from the Mesopotamian cultures, in- cluding religious texts, bureaucratic records, royal decrees, and more. Therefore they are a target of extensive transcription and transliteration efforts. One such transcription is exemplified by the La- tinized text to the right of the tablet in Figure 1. The Open Richly Annotated Cuneiform Corpus (Oracc)1 is one of the major Akkadian transcrip- tion collections, culminating in approximately 1http://oracc.org Figure 1: A clay tablet from Oracc (left) with its corre- sponding Latin transliteration (right). Words are delim- ited by spaces, while signs are delimited by hyphens or dots. A sign which is missing due to deterioration is de- noted by ‘x’ and highlighted in red in the figure. We de- velop models which automatically complete these miss- ing signs based on the surrounding context. 2.3M transcribed signs from 10K tablets. As fur- ther evidenced in Figure 1, many of the signs in the tablets were eroded over time and some parts were broken or lost, forcing editors to “fill in the gaps” where possible, based on the context of the surrounding words. In this paper, we identify that the task of masked language modeling, used ubiquitously in recent years for pretraining other downstream tasks (Pe- ters et al., 2018; Howard and Ruder, 2018; Liu et al., 2019) lends …

Content truncated to 500 words