Meaningfulness and unit of Zipf’s law: evidence from danmu comments

Meaningfulness and unit of Zipf’s law: evidence from danmu comments Yihan Zhou Department of East Asian Languages and Cultures University of Illinois at Urbana-Champaign 707 S Mathews Ave, Urbana, IL, USA 61801 yzhou114@illinois.edu Abstract Zipf’s law is a succinct yet powerful mathematical law in linguistics. However, the mean- ingfulness and units of the law have remained controversial. The current study uses online video comments call “danmu comment” to investigate these two questions. The results are consistent with previous studies arguing Zipf’s law is subject to topical coher- ence. Specifically, it is found that danmu comments sampled from a single video follow Zipf’s law better than danmu comments sampled from a collection of videos. The results also suggest the existence of multiple units of Zipf’s law. When different units including words, n-grams, and danmu comments are compared, both words and danmu comments obey Zipf’s law and words may be a better fit. The issues of combined n-grams in the literature are also discussed. 1 Introduction 1.1 Zipf’s law Zipf’s law is an important empirical law describing the statistical properties of many natural phenomena. The law states that the frequency of a word in a given corpus has an inverse proportion with its frequency rank (Zipf, 1949). Ideally, the word that ranks first will be twice as frequent as the word that ranks second and so forth. This quantitative relation is captured in the following equation where f represents word frequency, r stands for the frequency rank, and C is a constant. f = C r (1) Mandelbrot (1953) proposed a refinement of Zipf’s original equation by adding two constants m and β. m is the shifted rank and β is the exponent which is estimated to be 1. The following equation shows Mandelbrot’s revision: f = C (r + m)β (2) Zipf’s law is quite common in natural languages and language-related phenomena. It was found that the translated versions of the Holy Bible in one hundred natural languages approxi- mately follow Zipf’s law (Mehri and Jamaati, 2017). Zipf’s law also holds for artificial languages such as Esperanto, programming languages such as Python and UNIX (Chen, 1991; Manaris et al., 2006; Sano et al., 2012). Beyond languages, Zipf’s law also has a wide coverage in physical, biological, and behavioral phenomena. Examples include city sizes, webpage visits, scientific citation numbers, earthquake magnitudes (Cunha et al., 1995; Gabaix, 1999; Redner, 1998; Newman, 2005). ©2021 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 International License CCL 2021 Proceedings of the 20th China National Conference on Computational Linguistics, pages 1046-1057, Hohhot, China, Augest 13 - 15, 2021. (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China Computational Linguistics 1.2 Remaining questions in Zipf’s law Zipf’s law has been proposed for more than 70 years, yet a central question still persists: why the complex language production processes should conform to a mathematically concise equation (Piantadosi, 2014). While many studies have successfully demonstrated Zip’s law in languages, very few explained the …

Content truncated to 500 words