Segmentation-free compositional n-gram embedding

Proceedings of NAACL-HLT 2019, pages 3207–3215 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3207 Segmentation-free Compositional n-gram Embedding Geewook Kim and Kazuki Fukui and Hidetoshi Shimodaira Department of Systems Science, Graduate School of Informatics, Kyoto University Mathematical Statistics Team, RIKEN Center for Advanced Intelligence Project {geewook, k.fukui}@sys.i.kyoto-u.ac.jp, shimo@i.kyoto-u.ac.jp Abstract We propose a new type of representation learn- ing method that models words, phrases and sentences seamlessly. Our method does not depend on word segmentation and any human- annotated resources (e.g., word dictionaries), yet it is very effective for noisy corpora writ- ten in unsegmented languages such as Chinese and Japanese. The main idea of our method is to ignore word boundaries completely (i.e., segmentation-free), and construct representa- tions for all character n-grams in a raw cor- pus with embeddings of compositional sub-n- grams. Although the idea is simple, our exper- iments on various benchmarks and real-world datasets show the efficacy of our proposal. 1 Introduction Most existing word embedding models (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) take a sequence of words as their in- put. Therefore, the conventional models are de- pendent on word segmentation (Yang et al., 2017; Shao et al., 2018), which is a process of convert- ing a raw corpus (i.e., a sequence of characters) into a sequence of segmented character n-grams. After the segmentation, the segmented charac- ter n-grams are assumed to be words, and each word’s representation is constructed from distri- bution of neighbour words that co-occur together across the estimated word boundaries. However, in practice, this kind of approach has several prob- lems. First, word segmentation is difficult espe- cially when texts in a corpus are noisy or unseg- mented (Saito et al., 2014; Kim et al., 2018). For example, word segmentation on social network service (SNS) corpora, such as Twitter, is a chal- lenging task since it tends to include many mis- spellings, informal words, neologisms, and even emoticons. This problem becomes more severe in unsegmented languages, such as Chinese and Japanese, whose word boundaries are not explic- itly indicated. Second, word segmentation has am- biguities (Luo et al., 2002; Li et al., 2003). For example, a compound word 線形代数学 (linear algebra) can be seen as a single word or sequence of words, such as 線形|代数学 (linear | algebra). Word segmentation errors negatively influence subsequent processes (Xu et al., 2004). For exam- ple, we may lose some words in training corpora, leading to a larger Out-Of-Vocabulary (OOV) rate (Sun et al., 2005). Moreover, segmentation errors, such as segmenting きのう (yesterday) as き|のう (tree | brain), produce false co-occurrence information. This problem is crucial for most ex- isting word embedding methods as they are based on distributional hypothesis (Harris, 1954), which can be summarized as: “a word is characterized by the company it keeps” (Firth, 1957). To enhance word segmentation, some recent works (Junyi, 2013; Sato, 2015; Jeon, 2016) made rich resources publicly available. However, main- taining them up-to-date is difficult and …

Content truncated to 500 words