TRANSLIT: A Large-scale Name Transliteration Resource

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3265–3271 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3265 TRANSLIT: A Large-scale Name Transliteration Resource Fernando Benites, Gilbert François Duivesteijn, Pius von Däniken, Mark Cieliebak Zurich University of Applied Sciences, Deep Impact Switzerland benf@zhaw.ch, gilbert@deep-impact.ch, vode@zhaw.ch, ciel@zhaw.ch Abstract Transliteration is the process of expressing a proper name from a source language in the characters of a target language (e.g. from Cyrillic to Latin characters). We present TRANSLIT, a large-scale corpus with approx. 1.6 million entries in more than 180 languages with about 3 million variations of person and geolocation names. The corpus is based on various public data sources, which have been transformed into a unified format to simplify their usage, plus a newly compiled dataset from Wikipedia. In addition, we apply several machine learning methods to establish baselines for automatically detecting transliterated names in various languages. Our best systems achieve an accuracy of 92% on identification of transliterated pairs. Keywords: Transliteration of Names, Name Variant Discovery, Multi-lingual, Language Resource 1. Introduction Identifying named entities (e.g. persons or locations) is a crucial task in many important applications, from anti-money laundering to reputation monitoring to online surveillance of terrorism activities. If the texts are written in languages with different characters (e.g. Cyrillic, Arabic, Latin etc.), this becomes particularly challenging: names and places that origin from one language have often numer- ous different spellings in another language. For instance, the Russian name Горбачёв can be written in Latin characters as Gorbachev, Gorbachov, and Gorbachyov, and the same applies to many other names or geolocations. The task of transliteration focuses on the transformation of a word (normally a proper noun) into a language which has another phonology inventory and a different alphabet. Note that transliteration differs significantly from transcription, which focuses on the proper spelling of a foreign language sound into the target language. In principle, there exists a set of ISO1 rules which can be applied when converting a proper noun from a source lan- guage into a different target language. Unfortunately, they neither cover all language pairs (which is especially true for low-resources languages), nor are they always consistently applied. Note that these rules are most suited to be ap- plied manually, but not automatically. This creates a variety of (possible) transliterations2 which need to be considered when searching for entities. Thus, there is a strong need for large-scale corpora and high-quality methods to train auto- matic machine transliteration. Automatic Machine Transliteration There exist several approaches for automatic machine transliteration, and also some datasets on which automatic methods can be trained and evaluated (see Section 2 for more details). However, for most language-pairs there exist only little parallel data, or none at all. This is in particular 1https://www.iso.org/ics/01.140.10/x/ 2Therefore, we consider the task of transliteration/finding name variants in this study. We will use interchangeably name variant detection and name transliteration detection, as we con- sider name transliteration detection has an overlap with …

Content truncated to 500 words