Word Segmentation by Separation Inference for East Asian Languages

Findings of the Association for Computational Linguistics: ACL 2022, pages 3924 - 3934 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Word Segmentation by Separation Inference for East Asian Languages Yu Tong†, Jingzhi Guo†∗, Jizhe Zhou‡∗, Ge Chen§, Guokai Zheng§ †Department of Computer Science, University of Macau, Macau, China ‡Department of Computer Science, Sichuan University, China §vivo AI Lab, Shenzhen, China † ‡{yb87462, jzguo, yb87409}@umac.mo §{gechen.nlp, gkzheng.nlp}@gmail.com Abstract Chinese Word Segmentation (CWS) intends to divide a raw sentence into words through se- quence labeling. Thinking in reverse, CWS can also be viewed as a process of grouping a sequence of characters into a sequence of words. In such a way, CWS is reformed as a separation inference task in every adjacent character pair. Since every character is either connected or not connected to the others, the tagging schema is simplified as two tags "Con- nection" (C) or "NoConnection" (NC). There- fore, bigram is specially tailored for "C-NC" to model the separation state of every two con- secutive characters. Our Separation Inference (SpIn) framework is evaluated on five public datasets, is demonstrated to work for machine learning and deep learning models, and outper- forms state-of-the-art performance for CWS in all experiments. Performance boosts on Japanese Word Segmentation (JWS) and Ko- rean Word Segmentation (KWS) further prove the framework is universal and effective for East Asian Languages. 1 1 Introduction In Natural Language Processing (NLP), word seg- mentation is the commencement of Part-of-Speech (POS) tagging, semantic role labeling (SRL), and other similar studies. Particularly for Chinese, Japanese and Korean languages, the absence of explicit boundaries between characters makes the Word Segmentation (WS) task indispensable in NLP tasks. Dominant word segmentation methods considered WS as a sequence tagging task (Xue, 2003). Various tagging schemas such as "BMES" (Begin, Middle, End, Single), "BIES" (Begin, In- side, End, Single), "SEP-APP" (Separate, Append), "BI" (Begin, Inside), and "START-NONSTART" were employed to tackle the sequence labeling *Corresponding authors. 1Our source code will be released as soon as possible at https://github.com/UM-NLPer/SpIn-WS. task. These tagging schemas are all character- based and summarized as four-tags ("BMES", "BIES") and two-tags ("SEP-APP", "BI" "START- NONSTART"). Despite diverse tagging schemas, they all carry implicit position information. For four-tags tagging schemas, the implicit information restricts the transition between tags. Take "BMES" as an example; tag "B" can not be followed by "B" or "S". These two tagging schemas heavily rely on the precise prediction of the relative position of each character in one segment. However, the exact position information is not essential for the WS task. Any unreasonable inner prediction repre- senting the character’s relative position results in incorrect segmentation, although the correct bound- ary prediction. There is no limitation of tag-to-tag transition for the two-tags schema, but according to common sense, the first character of a sentence must be predicted as "SEP", "B" or "START". The implicit constraint of position for the first tag of the sentence still exists. It is necessary to ensure the prediction accuracy of the first tag during the …

Content truncated to 500 words