PyCantonese: Cantonese Linguistics and NLP in Python

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6607–6611 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6607 PyCantonese: Cantonese Linguistics and NLP in Python Jackson L. Lee, Litong Chen1, Charles Lam2, Chaak Ming Lau3, Tsz-Him Tsui 1Wheaton College, 2Hang Seng University of Hong Kong, 3Education University of Hong Kong Abstract This paper introduces PyCantonese, an open-source Python library for Cantonese linguistics and natural language processing. After the library design, implementation, corpus data format, and key datasets included are introduced, the paper provides an overview of the currently implemented functionality: stop words, handling Jyutping romanization, word segmentation, part-of-speech tagging, and parsing Cantonese text. Keywords: Cantonese, Jyutping, word segmentation, part-of-speech tagging, stop words 1. Introduction Cantonese is one of the most well-known Chinese va- rieties other than Mandarin Chinese. To handle Can- tonese language data and perform natural language pro- cessing (NLP) tasks, there did not exist an open-source software package dedicated to Cantonese. This pa- per introduces PyCantonese, an open-source Python li- brary for Cantonese linguistics and natural language processing.1 PyCantonese prioritizes ease of use in legal terms. It is publicly available for free for all purposes, including commercial ones. All of its dependent software pack- ages, datasets, and models have compatible licenses that allow free and public usage. The drawback is that only a very small number of Cantonese-specific datasets fulfill this requirement. While recent advances in NLP have been largely due to neural network-based machine learning coupled with the availability of a large amount of data, the fact that only a small amount of Cantonese data is legally available to PyCantonese means that it would be unrealistic for PyCantonese to train or include models based on neural networks. Al- though for this reason state-of-the-art NLP tools are not available, PyCantonese is useful for, among other use cases, creating more annotated Cantonese data, which can then be fed into other NLP systems. In the following, we first discuss the library design and implementation at a high level (section 2). We then ex- plain the corpus data format adopted by PyCantonese and introduce the datasets included (section 3). This is followed by an overview of the currently implemented functionality: stop words (section 4), Jyutping roman- ization (section 5), word segmentation (section 6), part- of-speech tagging (section 7), and parsing Cantonese text (section 8). 1Excluding the first author, all other authors are listed al- phabetically by last name. The version of PyCantonese dis- cussed in this paper is v3.4.0 released in December 2021. URL: https://pycantonese.org 2. Design and Implementation PyCantonese is built with a high level of usability and transparency in mind. The Python programming lan- guage is chosen for its popularity in general as well as in computational linguistics and NLP specifically. The software is readily available for download and installa- tion through the Python Package Index (i.e., by running pip install pycantonese). Major functional- ity is exposed as top-level functions for ease of access. The inputs to and outputs …

Content truncated to 500 words