A Hybrid Japanese Parser with Hand-crafted Grammar and Statistics

A Hybrid Japanese Parser with Hand-crafted Grammar and Statistics Hiroshi Kanayama†, Kentaro Torisawa‡∗, Yutaka Mitsuishi‡ and Jun’ichi Tsujii‡⋆ † Tokyo Research Laboratory, IBM Japan, Ltd. 1623-14 Shimo-tsuruma, Yamato-shi, Kanagawa 242-8502, Japan ‡ Department of Information Science, Graduate School of Science, University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan ∗ Information and Human Behavior, PRESTO, Japan Science and Technology Corporation Kawaguchi Hon-cho 4-1-8, Kawaguchi-shi, Saitama 332-0012, Japan ⋆ CCL, UMIST, U.K. {kanayama, torisawa, mitsuisi, tsujii}@is.s.u-tokyo.ac.jp Abstract This paper describes a hybrid parsing method for Japanese which uses both a hand-crafted gram- mar and a statistical technique. The key feature of our system is that in order to estimate likeli- hood for a parse tree, the system uses informa- tion taken from alternative partial parse trees gen- erated by the grammar. This utilization of alter- native trees enables us to construct a new statis- tical model called Triplet/Quadruplet Model. We show that this model can capture a certain ten- dency in Japanese syntactic structures and this point contributes to improvement of parsing accuracy on a shallow level. We report that, with an under- specified HPSG-based grammar and a maximum en- tropy estimation, our parser achieved high accuracy: 88.6% accuracy in dependency analysis of the EDR annotated corpus, and that it outperformed other purely statistical parsing methods on the same cor- pus. This result suggests that proper treatment of hand-crafted grammars can contribute to parsing ac- curacy on a shallow level. 1 Introduction There have been many attempts to combine hand- crafted high-level grammars, such as FB-LTAG, HPSG and LFG, and statistical disambiguation techniques to obtain precise linguistic structures (Schabes, 1992; Abney, 1996; Carroll et al., 1998). One evident advantage of this approach over purely statistical parsing techniques is that grammars can provide precise semantic representations. However, considering that remarkable parsing accuracy in a shallow level has been achieved by purely statisti- cal techniques (e.g. Ratnaparkhi (1997)), it may be thought more reasonable to use high-level grammars just for postprocessing which maps results of shallow syntactical analyses onto deep analyses. This work was conducted while the first author was a graduate student at Univ. of Tokyo. n h M NH H Figure 1: A tree M with a non-head daughter NH and a head daughter H. In this work we propose that hand-crafted high- level grammars can be useful in shallow-level analy- ses and statistical models. In our framework, gram- mars are used to obtain precise features for probabil- ity estimation, which are difficult to obtain without a grammar, and we show that such features contribute to high parsing accuracy on a shallow level. In this paper, the most preferable parse trees are chosen with a statistical model. In our method, the likelihood value L(M) of a (partial) tree M in Fig- ure 1 is defined as in (1): L(M) def = L(NH) × L(H) × P(n → h) (1) where NH is M’s non-head daughter (whose lexical head is n), H is the head-daughter (whose lexical head is h), and P(n …

Content truncated to 500 words