Reducing Lexical Features in Parsing by Word Embeddings

Reducing Lexical Features in Parsing by Word Embeddings Hiroya Komatsu, Ran Tian, Naoaki Okazaki and Kentaro Inui Tohoku University, Japan {h-komatsu, tianran, okazaki, inui}@ecei.tohoku.ac.jp Abstract The high-dimensionality of lexical features in parsing can be memory consuming and cause over-fitting problems. We propose a general framework to replace all lexical feature tem- plates by low-dimensional features induced from word embeddings. Applied to a near state-of-the-art dependency parser (Huang et al., 2012), our method improves the baseline, performs better than using cluster bit string features, and outperforms a recent neural net- work based parser. A further analysis shows that our framework has the effect hypothe- sized by Andreas and Klein (2014), namely (i) connecting unseen words to known ones, and (ii) encouraging common behaviors among in- vocabulary words. 1 Introduction Lexical features are powerful machine learning in- gredients for many NLP tasks, but the very high- dimensional feature space brought by these features can be memory consuming and cause over-fitting problems. Is it possible to use low-dimensional word embeddings to reduce the high-dimensionality of lexical features? In this paper, we propose a gen- eral framework for this purpose. As a proof of con- cept, we apply the framework to dependency pars- ing, since this is a task where lexical features are essential. Our approach is illustrated in Figure 1. Con- sider a transition-based dependency parser (Yamada and Matsumoto, 2003; Nivre et al., 2006; Zhang and Clark, 2008; Huang and Sagae, 2010; Zhang Templates: s0w Lexicon: … saw look … s0wsaw = (… 1 0 …) s0wlook = (… 0 1 …) q0w Lexicon: … you me … q0wyou = (… 1 0 …) q0wme = (… 0 1 …) … … ⋮ W(s0wsaw) W(s0wlook) ⋮ W(q0wyou) W(q0wme) ⋮ Weights: · = Scores Features: s0e1, … , s0ed q0e1, … , q0ed s0esaw = (0.6, … , 0.2) s0elook = (0.4, … , 0.3) q0eyou = (0.5, … , 0.8) q0eme = (0.7, … , 0.9) … … W(s0e1) ⋮ W(s0ed) W(q0e1) ⋮ W(q0ed) · = Scores Replace lexical feature templates by embedding features Figure 1: Each lexical feature template is replaced by a small number of embedding features. and Nivre, 2011), in which the words on top of the stack and the queue (denoted by s0w and q0w, re- spectively) are typically used as features to calcu- late scores of transitions. When s0w is used as a feature template, the features in this template (e.g. s0wsaw and s0wlook) can be viewed as one-hot vec- tors of a dimension of the lexicon size (Figure 1). Corresponding to s0w, a weight is assigned to each word (e.g. W(s0wsaw) and W(s0wlook)) for calcu- lating a transition score. Instead, we propose to utilize a d-dimensional word embedding, and re- place the feature template s0w by d features, namely s0e1, . . . , s0ed. Given the vector representation of a word (e.g., esaw = (0.6, . . . , 0.2)), we replace the lexical feature (e.g. s0wsaw) by a linear combination of the d features (e.g., …

Content truncated to 500 words