Sudachi: a Japanese Tokenizer for Business

Sudachi: a Japanese Tokenizer for Business Kazuma Takaoka†, Sorami Hisamoto†, Noriko Kawahara†, Miho Sakamoto†, Yoshitaka Uchida†, Yuji Matsumoto‡ †Works Applications ‡Nara Institute of Science and Technology {takaoka k hisamoto s, kawahara n, sakamoto mi, uchida yo}@worksap.co.jp, matsu@is.naist.jp Abstract Tokenization, or morphological analysis, is a fundamental and important technology for processing a Japanese text, especially for in- dustrial applications. However, we often face many obstacles, such as the inconsistency of token unit in different resources, notation variations, discontinued maintenance of the resources, and various issues with the existing tokenizer implementations. In order to im- prove this situation, we develop a tokenizer called Sudachi and its accompanying dictionary with features such as multi-granular output and normalization of notation variations. In addition to this, we continuously maintain our software and language resources in long-term as a part of the company business. We release the resulting tokenizer software and language resources freely available to the public as an open source software. You can access them at https://github.com/WorksApplications/Sudachi. Keywords: Tokenization, Morphological Analysis, Segmentation, Part-of-Speech Tagging, Lemmatization, Open Source Software 1. Introduction Unlike whitespace separation between words for English text, Japanese text does not contain explicit word bound- ary information. We need unobvious methods to recog- nize words within a text. For Japanese text, the process equivalent to “tokenization” in other languages is often called “morphological analysis”. It consists of three sub- processes, namely segmentation, part-of-speech (POS) tag- ging, and lemmatization (In the following we use the word “tokenization” to indicate these three processes). The definition of a token in Japanese is not trivial; People develop various systems and resources, each with different kinds of the standard. Lack of token unit compatibility is one of the critical problems of Japanese language resources. When we look at the applications of Japanese text process- ing in various business scenes, applying parsing or more advanced language process is not common. It is typical to just conduct tokenization and use its post-processed out- put. For many companies tokenization is a fundamental and important technology for text processing. However, when increasing number of companies are demanding Japanese text processing recently, we are lacking freely available and useful resources for tokenization. In order to improve this situation, we develop a new Japanese tokenizer and dictionary for business use. We make them available to the public as an open source soft- ware (OSS). 2. Previous Work 2.1. Japanese Tokenizers When conducting Japanese tokenization for business appli- cations, in the majority of cases MeCab1 (Kudo et al., 2004) or Kuromoji2 (the re-implementation of MeCab) are used. MeCab can process text at excellent speed, however, its functions are limited to segmentation, POS tagging, and 1http://taku910.github.io/mecab/ 2https://www.atilika.com/ja/kuromoji/ lemmatization; Users need to pre-/post-process the text by themselves. It is common to conduct text format- ting, sentence segmentation, and character normalization as pre-processing. Typical post-processing includes sim- ple chunking (e.g., for numeric expressions) and filtering by POS tags. Each user performs these processes on their own, therefore we tend to reinvent the wheel, or conduct such processes in …

Content truncated to 500 words