Segmentation-Free Word Embedding for Unsegmented Languages

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 767–772 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Segmentation-Free Word Embedding for Unsegmented Languages∗ Takamasa Oshikiri Graduate School of Engineering Science, Osaka University RIKEN Center for Advanced Intelligence Project CyberAgent, Inc. mail@oshikiri.org Abstract In this paper, we propose a new pipeline of word embedding for unsegmented lan- guages, called segmentation-free word em- bedding, which does not require word seg- mentation as a preprocessing step. Unlike space-delimited languages, unsegmented languages, such as Chinese and Japanese, require word segmentation as a prepro- cessing step. However, word segmenta- tion, that often requires manually anno- tated resources, is difficult and expensive, and unavoidable errors in word segmen- tation affect downstream tasks. To avoid these problems in learning word vectors of unsegmented languages, we consider word co-occurrence statistics over all pos- sible candidates of segmentations based on frequent character n-grams instead of segmented sentences provided by conven- tional word segmenters. Our experiments of noun category prediction tasks on raw Twitter, Weibo, and Wikipedia corpora show that the proposed method outper- forms the conventional approaches that re- quire word segmenters. 1 Introduction Word embedding, which learns dense vector rep- resentation of words from large text corpora, has received much attention in the natural language processing (NLP) community in recent years. It is reported that the representation of words well captures semantic and syntactic properties of words (Bengio et al., 2003; Mikolov et al., ∗ This work was done while the author was at Shi- modaira laboratory, Division of Mathematical Science, Grad- uate School of Engineering Science, Osaka University, and Mathematical Statistics Team, RIKEN Center for Advanced Intelligence Project. 20 10 0 10 20 20 10 0 10 20 blue MC BASARA K GANTZ MW I'll switch SNOW BLEACH PINK MONSTER REC YELLOW Free! DAYS C F RAY W3 DOLL AKIRA CLANNAD !! 10 � SS Wish X1 ! ReLIFE3 orange country ship food chemical compound profession railway station manga prefecture of Japan human Figure 1: t-SNE projections of vector represen- tation of Japanese nouns that generated by our proposed method without word dictionary. These proper nouns are color-coded according to its cat- egories which extracted from Wikidata. 2013; Pennington et al., 2014), and is useful for many downstream NLP tasks, including part-of- speech tagging, syntactic parsing, and machine translation (Huang et al., 2011; Socher et al., 2013; Sutskever et al., 2014). In order to train word embedding models on a raw text corpus, we have to do word segmentation as a preprocessing step. In space-delimited lan- guages such as English and Spanish, simple rule- based and co-occurrence-based approaches offer reasonable segmentations. On the other hands, these approaches are impractical for unsegmented languages such as Chinese, Japanese, and Thai. Therefore, machine learning-based approaches are widely used in NLP for unsegmented languages. Conditional random field (CRF)-based supervised word segmentation (Kudo et al., 2004; Tseng et al., 2005) is still the most used one in Japanese and Chinese NLP (Prettenhofer and Stein, 2010; …

Content truncated to 500 words