Word in a Dictionary is used by Numerous Users

International Joint Conference on Natural Language Processing, pages 874–877, Nagoya, Japan, 14-18 October 2013. Word in a Dictionary is used by Numerous Users Eiji Aramaki Sachiko Maskawa Kyoto University/PRESTO Photonic System Solutions eiji.aramaki@gmail.com sachiko.maskawa@gmail.com Mai Miyabe Mizuki Morita Sachi Yasuda Kyoto University University of Tokyo NINJAL mai.miyabe@gmail.com morita.mizuki@gmail.com yasudasac@gmail.com Abstract Dictionary editing requires enormous time to dis-cuss whether a word should be listed in a dictionary or not. So as to define a dictionary word, this study employs the number of word users as a novel metrics for selecting a dic- tionary word. In order to obtain the word user, we used about 0.25 billion tweets of approxi- mately 100,000 people published for five months. This study compared the classification performance of various measures. The result of the experiments revealed that a word in a dictionary is used by numerous users. 1 Introduction Dictionary editing requires numerous time to discuss whether a word should be listed in a dic- tionary or not. In order to define a dictionary word, this study assumes that the following two scales are essential than word frequency: (1) Usage period: a dictionary word has been used for lager period of time. (2) User population: a dictionary word has been used by more people. The first scale is hard to measure in practice, be- cause the usage period requires a longitudinal data. For the investigation, the second clue has more feasibility by social media resources, which enables to know the word usage for each user. The objective of this study is to retrieve a dictionary word. This study approaches this problem by drawing on a binary classification task, which divides the words into two catego- ries: a dictionary word (listed in a dictionary) and a out-of-dictionary word. For the database, we have collected 0.25 bil- lion tweets of 100,000 people from Twitter. The experimental results have revealed that a diction- ary word is highly correlated with the number of word users. Although the experiment is con- ducted in Japanese language, the proposed method does not depend on a specified language. 2 Related Work So far, a strong clue for dictionary editing is a word frequency. The relation between a word frequency and its coverage has been an interest for many researchers (Crowley 2003, Freeborn 2006, Burridge and Kortmann 2008). In English, the frequent 2,000 words cover 90% of spoken language (West 1953), and the most frequent 6,000 words cover 90% of written language (Francis, Kučera et al. 1982). The results in Jap- anese are similar to them. The frequent 10,000 words cover almost all vocabulary used in maga- zines (90 magazines) (NINJAL 1997), and 17,000 words cover the vocabulary spoken in television programs (Ishino 2000). Although the target media differs, they share the same findings that frequent words cover most of the corpus. This study presents another word measure. 3 Materials This study has used two types of data: user cor- pus (Section 3.1), and a gold standard data (Sec- tion 3.2): 3.1 Corpus: 100,000 people …

Content truncated to 500 words