Wikification for Scriptio Continua

Wikification for Scriptio Continua Yugo Murawaki, Shinsuke Mori {Graduate School of Informatics, Academic Center for Computing and Media Studies}, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan {murawaki, forest}@i.kyoto-u.ac.jp Abstract The fact that Japanese employs scriptio continua, or a writing system without spaces, complicates the first step of an NLP pipeline. Word segmentation is widely used in Japanese language processing, and lexical knowledge is crucial for reliable identification of words in text. Although external lexical resources like Wikipedia are potentially useful, segmentation mismatch prevents them from being straightfor- wardly incorporated into the word segmentation task. If we intentionally violate segmentation standards with the direct incorporation, quantitative evaluation will be no longer feasible. To address this problem, we propose to define a separate task that directly links given texts to an external resource, that is, wikification in the case of Wikipedia. By doing so, we can circumvent segmentation mismatch that may not necessarily be important for downstream applications. As the first step to realize the idea, we design the task of Japanese wikification and construct wikification corpora. We annotated subsets of the Balanced Corpus of Contemporary Written Japanese plus Twitter short messages. We also implement a simple wikifier and investigate its performance on these corpora. Keywords: corpus annotation, wikification, word segmentation 1. Introduction Wikification is the task of detecting concept mentions in text and disambiguating them into their corresponding Wikipedia articles. It was originally introduced as an infor- mation retrieval task (Mihalcea and Csomai, 2007; Ferrag- ina and Scaiella, 2010), and then has found a wide range of NLP applications including measuring semantic related- ness between text fragments (Gabrilovich and Markovitch, 2007), text classification (Chang et al., 2008), and corefer- ence resolution (Ratinov and Roth, 2012). In this paper, we present new Japanese corpora for wikifica- tion. A major difference from previous studies on English lies in the fact that like Chinese and Thai, Japanese em- ploys scriptio continua, or a writing system in which words are not delimited by white space. For this reason, word segmentation is widely used as a prerequisite for most NLP tasks, and wikification could be another such task. How- ever, the pipeline architecture often suffers from error prop- agation. Indeed, our motivation for Japanese wikification comes not only from the above-mentioned applications but from the lexical bottleneck problem in word segmentation. While decades of research in Japanese word segmentation has achieved a reasonable level of accuracy in benchmark cor- pora (Kudo et al., 2004), it is still prone to errors when it comes to real-world texts. One of the key problems is so- called unknown words, or the lack of lexical knowledge. In essence, in order to reliably identify words in text, the segmenter needs to know them in advance, by representing them directly as dictionary items (Kurohashi et al., 1994; Asahara and Matsumoto, 2000; Kudo et al., 2004) or indi- rectly as corpus-induced features (Neubig et al., 2011). Al- though lexical knowledge has been maintained by experts, it is impractical to adapt it to web-scale …

Content truncated to 500 words