Simplified Corpus with Core Vocabulary

Simplified Corpus with Core Vocabulary Takumi Maruyama, Kazuhide Yamamoto Nagaoka University of Technology 1603-1, Kamitomioka Nagaoka, Niigata 940-2188, JAPAN {maruyama, yamamoto}@jnlp.org Abstract We have constructed the simplified corpus for the Japanese language and selected the core vocabulary. The corpus has 50,000 manually simplified and aligned sentences. This corpus contains the original sentences, simplified sentences and English translation of the original sentences. It can be used for automatic text simplification as well as translating simple Japanese into English and vice-versa. The core vocabulary is restricted to 2,000 words where it is selected by accounting for several factors such as meaning preservation, variation, simplicity and the UniDic word segmentation criterion. We repeated the construction of the simplified corpus and, subsequently, updated the core vocabulary accordingly. As a result, despite vocabulary restrictions, our corpus achieved high quality in grammaticality and meaning preservation. In addition to representing a wide range of expressions, the core vocabulary’s limited number helped in showing similarities of expressions among simplified sentences. We believe that the same quality can be obtained by extending this corpus. Keywords: Corpus, Controlled Languages, Lexicon 1. Introduction Over the years, the number of foreigners visiting Japan has been increasing. Japan hosts around 24 million visitors in a year. In addition, there are about 2.47 million foreign residents in Japan, and this number is also increasing. According to a survey conducted by the National Insti- tute for Japanese Language and Linguistics, only 44.0% of Japan’s foreign residents can speak English (Iwata, 2010). This ratio is lower than the percentage of people who can speak Japanese (62.6%). Foreigners can understand simple Japanese more easily than English. Therefore, we need to consider simple Japanese as a means of providing informa- tion for foreigners. Simple Japanese is the language with less complexity of vocabulary, grammar, and expression. This makes it possible to provide many text resources to a wide range of readers including Japan’s foreign residents, foreign tourists, children, and intellectually disabled peo- ple. We have been researching text simplification for several years (Moku et al., 2012; Kajiwara and Yamamoto, 2013; Kajiwara and Yamamoto, 2015). In this paper, we focus on vocabulary size because it can be defined objec- tively. There is a gap between the vocabulary size necessary for understanding the media and the vocab- ulary size necessary for understanding basic Japanese. According to a survey in modern Japanese magazines, 12,000 words are required to practically use Japanese (Tamamura, 2002). In addition, in order to understand TV shows sufficiently, it is necessary to know 17,000 words (National Institute Japanese Language and Linguistics , 1999). On the other hand, according to the standard of the Japanese Language Proficiency Test (called JLPT) Level 3 (level of understanding elementary Japanese), it is necessary to master 1,500 words. Moreover, Japanese vocabulary size essential for daily life is considered to be about 1,000 to 2,000 words (Kai, 2002). We think that eliminating this gap helps to understand the Japanese language. We manually rewrote sentences which were extracted from newspaper articles and broadcast media …

Content truncated to 500 words