Building and Annotating the Linguistically Diverse NTU-MC (NTU-Multilingual Corpus)

Copyright 2011 by Tan Liling and Francis Bond. Building and Annotating the Linguistically Diverse NTU-MC (NTU– Multilingual Corpus)* Liling Tan and Francis Bond Division of Linguistics and Multilingual Studies, Nanyang Technological University, 14 Nanyang Drive, Singapore, 637332, Singapore alvations@gmail.com, bond@ieee.org Abstract. The NTU-MC compilation taps on the linguistic diversity of multilingual texts available within Singapore. The current version of NTU-MC contains 375,000 words (15,000 sentences) in 6 languages (English, Chinese, Japanese, Korean, Indonesian and Vietnamese) from 6 language families (Indo-European, Sino-Tibetan, Japonic, Korean as a language isolate, Austronesian and Austro-Asiatic). The NTU-MC is annotated with a layer of monolingual annotation (POS tags) and cross-lingual annotation (sentence-level alignments). The diverse language data and cross-lingual annotations provide valuable information on linguistic diversity for traditional linguistic research as well as natural language processing tasks. This paper describes the corpus compilation process with the evaluation of the monolingual and cross-lingual annotations of the corpus data. The corpus is available under the Creative Commons – Attribute 3.0 Unported license (CC by). Keywords: Multilingual, Corpus, Parallel text * The authors of this paper thank Ms Joan Lee, New Media Manager of the Singapore Tourist Board, for granting permission to use and redistribute text from their multilingual websites (www.yoursingapore.com.sg). This research was partially funded by a joint JSPS/NTU grant on Revealing Meaning through Multiple Languages. 1 Introduction “The rapidly growing gap between the demand for high-quality multilingual content and the lag in the supply of language professionals is driving the requirement for technology that can dramatically improve translation turnaround time while maintaining exceptionally high output quality" (McCallum, 2011). Cross-lingual training using parallel corpora has been gaining popularity in NLP application tasks such as word sense disambiguation (e.g. Sarrafzadeh et al. 2011; Saravanan et al. 2010; Mitamura et al. 2007), information retrieval and question- answering. In addition, parallel corpora are valuable resources for advancing linguistic annotations morphologically, syntactically and semantically (e.g. Snyder and Barzilay; 2008, Hwa et al. 2005; Resnik, 2004). The essential knowledge resource in building these language technologies are grounded on parallel corpora. The present pool of resources holds a sizable amount of European parallel corpora (e.g. Ralf et al. 2006; Erjavec, 2004), an increasing interest in building Asian languages-English bitexts (e.g. Xiao, 2004) but only a handful of parallel Asian language corpora (e.g. Zhang, 2005). To fill the lack of parallel corpora of Asian languages, the NTU–Multilingual Corpus (NTU- MC) taps on the array of multilingual texts available in Singapore; ranging from the multilingual sign boards with official languages of Singapore (English, Chinese, Malay, Tamil) to posters, signs and guides targeted towards migrants, expats and tourists (in Indonesian, Japanese, Korean, Vietnamese, Thai, Tagalog, etc.). Singapore‟s multicultural and multilingual 25th Pacific Asia Conference on Language, Information and Computation, pages 362–371 362 society has necessitated the use of parallel text for signboards, public announcements and information dissemination. The NTU-MC presents multilingual data from a modern cosmopolitan city where people interact in different languages. Empirically, the NTU-MC represents unique societal linguistic diversity; computationally, the NTU-MC provides diverse parallel …

Content truncated to 500 words