Orthographic Syllable as basic unit for SMT between Related Languages

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1912–1917, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Orthographic Syllable as basic unit for SMT between Related Languages Anoop Kunchukuttan, Pushpak Bhattacharyya Center For Indian Language Technology, Department of Computer Science & Engineering Indian Institute of Technology Bombay {anoopk,pb}@cse.iitb.ac.in Abstract We explore the use of the orthographic syl- lable, a variable-length consonant-vowel se- quence, as a basic unit of translation between related languages which use abugida or alpha- betic scripts. We show that orthographic sylla- ble level translation significantly outperforms models trained over other basic units (word, morpheme and character) when training over small parallel corpora. 1 Introduction Related languages exhibit lexical and structural sim- ilarities on account of sharing a common ances- try (Indo-Aryan, Slavic languages) or being in pro- longed contact for a long period of time (Indian sub- continent, Standard Average European linguistic ar- eas) (Bhattacharyya et al., 2016). Translation be- tween related languages is an important requirement due to substantial government, business and social communication among people speaking these lan- guages. However, most of these languages have few parallel corpora resources, an important requirement for building good quality SMT systems. Modelling the lexical similarity among related languages is the key to building good-quality SMT systems with limited parallel corpora. Lexical sim- ilarity implies that the languages share many words with the similar form (spelling/pronunciation) and meaning e.g. blindness is andhapana in Hindi, aandhaLepaNaa in Marathi. These words could be cognates, lateral borrowings or loan words from other languages. Translation for such words can be achieved by sub-word level transformations. For in- stance, lexical similarity can be modelled in the stan- dard SMT pipeline by transliteration of words while decoding (Durrani et al., 2010) or post-processing (Nakov and Tiedemann, 2012; Kunchukuttan et al., 2014). A different paradigm is to drop the notion of word boundary and consider the character n-gram as the basic unit of translation (Vilar et al., 2007; Tiedemann, 2009a). Such character-level SMT bas been explored for closely related languages like Bulgarian-Macedonian, Indonesian-Malay with modest success, with the short context of unigrams being a limiting factor (Tiedemann, 2012). The use of character n-gram units to address this limi- tation leads to data sparsity for higher order n-grams and provides little benefit (Tiedemann and Nakov, 2013). In this work, we present a linguistically moti- vated, variable length unit of translation — ortho- graphic syllable (OS) — which provides more con- text for translation while limiting the number of ba- sic units. The OS consists of one or more conso- nants followed by a vowel and is inspired from the akshara, a consonant-vowel unit, which is the funda- mental organizing principle of Indic scripts (Sproat, 2003; Singh, 2006). It can be thought of as an ap- proximate syllable with the onset and nucleus, but no coda. While true syllabification is hard, ortho- graphic syllabification can be easily done. Atreya et al. (2016) and Ekbal et al. (2006) have shown that the …

Content truncated to 500 words