A Fast and Accurate Partially Deterministic Morphological Analysis

Proceedings of Recent Advances in Natural Language Processing, pages 804–809, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_093 804 A Fast and Accurate Partially Deterministic Morphological Analysis Hajime Morita Tomoya Iwakura Fujitsu Laboratories Ltd., Kanagawa, Japan {hmorita, iwakura.tomoya}@fujitsu.com Abstract This paper proposes a partially determin- istic morphological analysis method for improved processing speed. Maximum matching is a fast deterministic method for morphological analysis. However, the method tends to decrease performance due to lack of consideration of contex- tual information. In order to use maxi- mum matching safely, we propose the use of Context Independent Strings (CISs), which are strings that do not have am- biguity in terms of morphological analy- sis. Our method first identifies CISs in a sentence using maximum matching with- out contextual information, then analyzes the unprocessed part of the sentence us- ing a bi-gram-based morphological anal- ysis model. We evaluate the method on a Japanese morphological analysis task. The experimental results show a 30% reduc- tion of running time while maintaining im- proved accuracy. 1 Introduction Morphological analysis segments a text into a se- quence of morphemes. The analysis is the first step of analyzing languages such as Chinese, Thai and Japanese. In Japanese, the analysis is a com- pound task of word segmentation, POS tagging, recognition of conjugation and lemmatization. Japanese morphological analysis derives a word dictionary. The dictionary is a set of tuples of word surface (conjugated form), base form (lemma of words) and POS tags. Given an input string of a sentence, the morphological analyzer enumerates all substrings of the sentence that are listed in the dictionary. Then the analyzer builds a large lattice (DAG) of the words as a set of candidate word se- quences. Because the lattice often becomes large, the lattice building tends to be a bottle neck of the morphological analysis. Finally, the analyzer se- lects a path on the lattice and gets the analysis re- sult. Morphological analysis is often used as a pre- processing step for shallow analyses on a large amount of texts collected from Web. These analy- ses include word counting, keyword extraction and named entity recognition. Morphological analy- sis tends to occupy a larger part of the processing time in the tasks. Therefore, we investigated a new morphological analysis method that improves effi- ciency and performance. There are three kinds of Japanese morpho- logical analysis methods in terms of the range of using contexts. The first group is maximum matching-based methods that do not use context (Sproat et al., 1996; Nagata, 1997; Sassano, 2014). Maximum matching is a significantly fast deterministic method for morphological analysis. However, this method tends to have a low performance, because it often fails an segmentation of phrases which have ambiguity depending on their context. The second group is neural network based methods that do not have explicit limit to the range of context (Morita et al., 2015; Chen et al., 2015; Wang et al., 2016; Kurita et al., 2017; Tolmachev et al., 2018). How- ever, neural network based methods generally require …

Content truncated to 500 words