Construction and Analysis of a Large Vietnamese Text Corpus

Construction and Analysis of a Large Vietnamese Text Corpus Dieu-Thu Le, Uwe Quasthoff IMS - University of Stuttgart, NLP Department - University of Leipzig dieu-thu.le@ims.uni-stuttgart.de, quasthoff@informatik.uni-leipzig.de Abstract This paper presents a new Vietnamese text corpus which contains around 4.05 billion words. It is a collection of Wikipedia texts, newspaper articles and random web texts. The paper describes the process of collecting, cleaning and creating the corpus. Processing Vietnamese texts faced several challenges, for example, different from many Latin languages, Vietnamese language does not use blanks for separating words, hence using common tokenizers such as replacing blanks with word boundary does not work. A short review about different approaches of Vietnamese tokenization is presented together with how the corpus has been processed and created. After that, some statistical analysis on this data is reported including the number of syllable, average word length, sentence length and topic analysis. The corpus is integrated into a framework which allows searching and browsing. Using this web interface, users can find out how many times a particular word appears in the corpus, sample sentences where this word occurs, its left and right neighbors. Keywords: corpus construction, text preprocessing, Vietnamese, topic modeling, searching, word co-occurrences 1. Introduction Vietnamese text processing started to become active about twelve years ago. Since then, several corpora have been built for some specific natural language processing tasks. (Pham et al., 2007) presented a corpus consisting of news- papers coming from two news sources collected within 6 months in 2005. This corpus was annotated with entity classes such as person, location, organization for named entity recognition. (Tu et al., 2006) released a corpus of 305 newspaper articles together with a list of 2,000 personal names and 707 locations.1 (Do et al., 2009) prepared a par- allel corpus of Vietnamese-French consisting of around 12 million (M) document pairs for machine translation. The EVC corpus produced at the national university of Ho Chi Minh (VNUHCM) consists of 400,000 pairs of Englist- Vietnamese sentences with approximately 5,500,000 words in the fields of Science and Technology. (Pham et al., 2008) presented a corpus of 1 million words collected from news- papers and children’s literature for a comparison in the word uses in children’s literature and in general text. SEAlang Li- brary Vietnamese Text Corpus introduced a corpus search interface with word neighbors and sample sentences includ- ing more than 79M characters.2 The Vietnamese corpus project3 also collected data from newspapers and annotate them with information such as author, public date, register date. In comparison to other available Vietnamese news corpora, this collection is one of the most comprehensive corpora containing a large amount of text collected from various sources. It can serve as a resource for different Vietnamese natural language processing tasks and text analysis. 1http://www.jaist.ac.jp/ hieuxuan/vnwordseg/data/ 2http://sealang.net/vietnamese/corpus.htm 3http://pers-www.wlv.ac.uk/ĩn6930/corpus.htm 2. The construction of the Vietnamese corpus 2.1. Data sources The Vietnamese corpus is a Web corpus collected between 2007 and 2014. It contains about 70 million of sentences with about 4.05 billion running words. The main sources …

Content truncated to 500 words