SansTib, a Sanskrit - Tibetan Parallel Corpus and Bilingual Sentence Embedding Model

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6728–6734 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6728 SansTib, a Sanskrit - Tibetan Parallel Corpus and Bilingual Sentence Embedding Model Sebastian Nehrdich Institute for Language and Information,Heinrich-Heine-Universit¨at D¨usseldorf; Khyentse Center for Tibetan Buddhist Textual Scholarship, Universit¨at Hamburg nehrdich@uni-duesseldorf.de Abstract This paper presents the development of SansTib, a Sanskrit - Classical Tibetan parallel corpus automatically aligned on sentence-level, and a bilingual sentence embedding model. The corpus has a size of about 317,289 sentence pairs and 14,420,771 tokens and thereby is a considerable improvement over previous resources for these two languages. The data is incorporated into the BuddhaNexus database to make it accessible to a larger audience. It also presents a gold evaluation dataset and assesses the quality of the automatic alignment. Keywords: Sanskrit, Tibetan, Sentence Alignment, Bilingual Sentence Representation, Parallel Corpus 1. Introduction Translations of Buddhist texts have been of central im- portance to the formation of the Tibetan Buddhist lit- erary tradition. The majority of these translations are from Indian Buddhist texts, of which many have been composed in Sanskrit or Middle Indic languages. The study of these Tibetan translations is not only very im- portant for the study of the Tibetan Buddhist tradi- tion, but also for the study of Buddhism in general, since many Indian Buddhist texts did not survive in their original language. Digitally available parallel data therefore can facilitate the research on Tibetan trans- lations of Sanskrit texts substantially. While consid- erable effort has been undertaken by the community of scholars to create large bilingual dictionaries for the two languages,1 a comprehensive parallel corpus with sentence alignment is a desideratum that this pa- per seeks to address. Both Sanskrit and Classical Tibetan are low-resource languages that still require a lot of linguistic research. In recent years, comparatively large digital monolin- gual corpora of Sanskrit and Tibetan Buddhist litera- ture have become available.2 At the same time, sen- tence alignment algorithms supported by multilingual sentence embedding have reached increasing levels of precision for noisy texts of low resource languages (Thompson and Koehn, 2019). It is now possible to automatically align Sanskrit texts with their respective Tibetan translations and reach a sufficient level of pre- cision to use these aligned texts either as a direct re- source for philological research in the form of an on- line database3 or for the training of multilingual lan- 1See for example the Tibetan-Sanskrit dictionary by J. Negi (Negi, 1993 2005). 2See section 4 for examples of such monolingual re- sources. 3The results are accessible at https:// buddhanexus.net/multi/neutral. guage models and neural machine translation models (see Thompson and Koehn (2019), section 4.3). This paper makes the following contributions to this prob- lem: • A sentence-level aligned parallel corpus of San- skrit Buddhist texts and their Tibetan translations with a total number of 317,289 sentence pairs, which is much larger and covers a greater variety of domains than the already available bilingual re- …

Content truncated to 500 words