Corpus Effects on the Evaluation of Automated Transliteration Systems

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 640–647, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Corpus Effects on the Evaluation of Automated Transliteration Systems Sarvnaz Karimi Andrew Turpin Falk Scholer School of Computer Science and Information Technology RMIT University, GPO Box 2476V, Melbourne 3001, Australia {sarvnaz,aht,fscholer}@cs.rmit.edu.au Abstract Most current machine transliteration sys- tems employ a corpus of known source- target word pairs to train their system, and typically evaluate their systems on a similar corpus. In this paper we explore the perfor- mance of transliteration systems on corpora that are varied in a controlled way. In partic- ular, we control the number, and prior lan- guage knowledge of human transliterators used to construct the corpora, and the origin of the source words that make up the cor- pora. We find that the word accuracy of au- tomated transliteration systems can vary by up to 30% (in absolute terms) depending on the corpus on which they are run. We con- clude that at least four human transliterators should be used to construct corpora for eval- uating automated transliteration systems; and that although absolute word accuracy metrics may not translate across corpora, the relative rankings of system performance re- mains stable across differing corpora. 1 Introduction Machine transliteration is the process of transform- ing a word written in a source language into a word in a target language without the aid of a bilingual dictionary. Word pronunciation is preserved, as far as possible, but the script used to render the target word is different from that of the source language. Transliteration is applied to proper nouns and out- of-vocabulary terms as part of machine translation and cross-lingual information retrieval (CLIR) (Ab- dulJaleel and Larkey, 2003; Pirkola et al., 2006). Several transliteration methods are reported in the literature for a variety of languages, with their per- formance being evaluated on multilingual corpora. Source-target pairs are either extracted from bilin- gual documents or dictionaries (AbdulJaleel and Larkey, 2003; Bilac and Tanaka, 2005; Oh and Choi, 2006; Zelenko and Aone, 2006), or gathered ex- plicitly from human transliterators (Al-Onaizan and Knight, 2002; Zelenko and Aone, 2006). Some eval- uations of transliteration methods depend on a single unique transliteration for each source word, while others take multiple target words for a single source word into account. In their work on transliterating English to Persian, Karimi et al. (2006) observed that the content of the corpus used for evaluating systems could have dramatic affects on the reported accuracy of methods. The effects of corpus composition on the evalua- tion of transliteration systems has not been specif- ically studied, with only implicit experiments or claims made in the literature such as introduc- ing the effects of different transliteration mod- els (AbdulJaleel and Larkey, 2003), language fam- ilies (Lind´en, 2005) or application based (CLIR) evaluation (Pirkola et al., 2006). In this paper, we re- port our experiments designed to explicitly examine the effect that varying the underlying corpus used in …

Content truncated to 500 words