Keyboard Logs as Natural Annotations for Word Segmentation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1186–1196, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Keyboard Logs as Natural Annotations for Word Segmentation Fumihiko Takahashi∗ Yahoo Japan Corporation Midtown Tower, 9-7-1 Akasaka, Minato-ku, Tokyo, Japan ftakahas@yahoo-corp.jp Shinsuke Mori ACCMS, Kyoto University Yoshida Honmachi, Sakyo-ku, Kyoto, Japan forest@i.kyoto-u.ac.jp Abstract In this paper we propose a framework to improve word segmentation accuracy us- ing input method logs. An input method is software used to type sentences in lan- guages which have far more characters than the number of keys on a keyboard. The main contributions of this paper are: 1) an input method server that proposes word candidates which are not included in the vocabulary, 2) a publicly usable input method that logs user behavior (like typ- ing and selection of word candidates), and 3) a method for improving word segmen- tation by using these logs. We conducted word segmentation experiments on tweets from Twitter, and showed that our method improves accuracy in this domain. Our method itself is domain-independent and only needs logs from the target domain. 1 Introduction The first step of almost all natural language processing (NLP) for languages with ambiguous word boundaries (such as Japanese and Chinese) is solving the problem of word identification am- biguity. This task is called word segmentation (WS) and the accuracy of state-of-the-art methods based on machine learning techniques is more than 98% for Japanese and 95% for Chinese (Neubig et al., 2011; Yang and Vozila, 2014). Compared to languages like English with clear word bound- aries, this ambiguity poses an additional problem for NLP tasks in these languages. To make matters worse, the domains of the available training data often differ from domains where there is a high demand for NLP, which causes a severe degrada- tion in WS performance. Examples include ma- *This work was done when the first author was at Kyoto University. chine translation of patents, text mining of med- ical texts, and marketing on the micro-blog site Twitter1. Some papers have reported low accuracy on WS or the joint task of WS and part-of-speech (POS) tagging of Japanese or Chinese in these do- mains (Mori and Neubig, 2014; Kaji and Kitsure- gawa, 2014; Liu et al., 2014) To cope with this problem, we propose a way to collect information from people as they type Japanese or Chinese on computers. These lan- guages use far more characters than the number of keys on a keyboard, so users use software called an input method (IM) to type text in these languages. Unlike written texts in these languages, which lack word boundary information, text entered with an IM can provide word boundary information that can used by NLP systems. As we show in this pa- per, logs collected from IMs are a valuable source of word boundary information. An IM consists of a client (front-end) and a server (back-end). The client receives a key se- quence typed by the user and sends …

Content truncated to 500 words