Integration of Reduplicated Multiword Expressions and Named Entities in a Phrase Based Statistical Machine Translation System

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1304–1312, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Integration of Reduplicated Multiword Expressions and Named Entities in a Phrase Based Statistical Machine Translation System Thoudam Doren Singh† Department of Computer Science and Engineering Jadavpur University Kolkata-700032, India thoudam.doren@gmail.com Sivaji Bandyopadhyay Department of Computer Science and Engineering Jadavpur University Kolkata-700032, India sivaji_cse_ju@yahoo.com Abstract The language specific Multiword expressions (MWEs) play important roles in many natural language processing (NLP) tasks. Integrating reduplicated multiword expressions (RMWEs) into the Phrase Based Statistical Machine Translation (PBSMT) to improve translation quality is reported in the present work between Manipuri, a highly agglutinative Tibeto-Burman language and English. In addition, Multiword Named Entities (MNEs) coupled with Transliterated non-named entities (non-NE) between Manipuri and English phrase based SMT system are also integrated. The tighter integration of RMWEs and NEs into the PBSMT is carried out after automatic extraction using SVM based machine learning technique followed by automatic bilingual RMWE and MNE extraction using GIZA++ alignment. Our experimental results show improvement in the PBSMT system BLEU and NIST scores over the baseline system. Subjective evaluation indicates the improvement in the adequacy. 1 Introduction Multiword expressions (MWEs) are a key challenge for the development of large-scale, linguistically sound natural language processing technology (Sag et al, 2002). The various kinds of multiword expressions should be analyzed in distinct ways. An adequate comprehensive analysis of multiword expressions must employ both symbolic and statistical techniques. MWEs span a range of constructions, from completely frozen, semantically opaque idiomatic expressions, to frequent but morphologically productive and semantically compositional collocations. Various linguistic processes (orthographic, morphological, syntactic, semantic and cognitive) apply to MWEs in idiosyncratic ways. Notably, MWEs blur the distinction between the lexicon and the grammar, since they often have some properties of words and some properties of phrases. The MWE identification works concentrate on compound nouns, noun-verb combination, idioms and phrases for several languages such as Hindi and Hebrew but not much on RMWEs. The reason may be that the reduplicated words are either rare or easy to identify for these languages since only complete duplication and some amount of partial reduplication may be present in these languages. On the other hand, reduplicated MWEs (RMWE) of several varieties are widely present in Manipuri1. In the present work, the identification of Manipuri Named Entities (NEs) and the identification and classification of RMWEs is carried out using the SVM based machine learning approach. † Presently at Center for Development of Advanced Computing (CDAC), Mumbai, India 1 Manipuri, locally known as Meiteilon or Meeteilon, is a less privileged, morphologically rich, highly agglutinative language spoken basically in the states of Manipur, Assam, Tripura and Mizoram in India and in the neighboring countries of Myanmar and Bangladesh approximately by three million speakers. Manipuri became the first Tibeto-Burman (TB) language to receive recognition in the year 1992 as a schedule VIII language of India. 1304 Manipuri is very rich in RMWEs like other Tibeto-Burman languages. The work …

Content truncated to 500 words