Observing Features of PTT Neologisms: A Corpus-driven Study with N-gram Model

Observing Features of PTT Neologisms: A Corpus-driven Study with N-gram Model Tsun-Jui Liu Graduate Institute of Linguistics National Taiwan University r99142008@ntu.edu.tw Shu-Kai Hsieh Graduate Institute of Linguistics National Taiwan University shukaihsieh@ntu.edu.tw Laurent PREVOT Laboratoire Parole et Langage Université Aix-Marseille laurent.prevot@lpl-aix.fr Abstract PTT (批踢踢) is one of the largest web forums in Taiwan. In the last few years, its importance has been growing rapidly because it has been widely mentioned by most of the mainstream media. It is observed that its influence reflects not only on the society but also on the language novel use in Taiwan. In this research, a pipeline processing system in Python was developed to collect the data from PTT, and the n-gram model with proposed linguistic filter are adopted with the attempt to capture two-character neologisms emerged in PTT. Evaluation task with 25 subjects was conducted against the system's performance with the calculation of Fleiss’ kappa measure. Linguistic discussion as well as the comparison with time series analysis of frequency data are provided. It is hoped that the detection of neologisms in PTT can be improved by observing the features, which may even facilitate the prediction of the neologisms in the future. Keywords: PTT, Neologisms, n-gram, Fleiss’ kappa, Time series analysis 1. Introduction A neologism in general refers to “a newly coined term, word, or phrase, that may be in the process of entering common use, but has not yet been accepted into mainstream language” (Levchenko, 2010)1. It is closely related to the unknown words or out-of-vocabulary in the field of Speech and Natural Language Processing, but with the nuance that the latter is often formally defined by its non-existence in a given vocabulary repository. With the emergence of voluminous data on the web and fast-developing technologies, never before has our world been facing with such an overwhelming mass of neologisms. Therefore, the description and 1 As cited by wiki at http://en.wikipedia.org/wiki/Neologism#cite_ref-1 Proceedings of the Twenty-Fifth Conference on Computational Linguistics and Speech Processing (ROCLING 2013) 250 detection of neologism has become an important research topic in the recent years. In this paper, we aim to begin with a corpus-driven approach in exploring the linguistic features of Chinese neologisms. We use PTT as our corpus data. As widely known, PTT is one of the largest web forums in Taiwan that contain users from various backgrounds and ages. In these years, its importance has been growing rapidly because it has been widely mentioned by most of the mainstream media in Taiwan. As Magistry (2012) suggested, “PTT should be seen as an extension of the modern society in Taiwan.” This implies that PTT has great influence not only on the society but also the novel language use in Taiwan, which motives this research to exploit PTT as data source. Section 2 explains the pipeline framework developed for data crawling and pre-processing, and the lexicon and filter for capturing two-character neologisms in PTT. Section 3 introduces the methodological part, where the rationale of our proposed ‘diachronic n-gram model’ is introduced …

Content truncated to 500 words