Embedding Lexical Features via Low-Rank Tensors

Proceedings of NAACL-HLT 2016, pages 1019–1029, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Embedding Lexical Features via Low-Rank Tensors Mo Yu∗ Harbin Institute of Technology IBM Watson yum@us.ibm.com Mark Dredze HLTCOE Johns Hopkins University mdredze@cs.jhu.edu Raman Arora Johns Hopkins University arora@cs.jhu.edu Matthew R. Gormley Carnegie Mellon University mgormley@cs.cmu.edu Abstract Modern NLP models rely heavily on engi- neered features, which often combine word and contextual information into complex lexi- cal features. Such combination results in large numbers of features, which can lead to over- fitting. We present a new model that repre- sents complex lexical features—comprised of parts for words, contextual information and labels—in a tensor that captures conjunction information among these parts. We apply low- rank tensor approximations to the correspond- ing parameter tensors to reduce the parame- ter space and improve prediction speed. Fur- thermore, we investigate two methods for han- dling features that include n-grams of mixed lengths. Our model achieves state-of-the-art results on tasks in relation extraction, PP- attachment, and preposition disambiguation. 1 Introduction Statistical NLP models usually rely on hand- designed features, customized for each task. These features typically combine lexical and contextual in- formation with the label to be scored. In relation extraction, for example, there is a parameter for the presence of a specific relation occurring with a fea- ture conjoining a word type (lexical) with depen- dency path information (contextual). In measur- ing phrase semantic similarity, a word type is con- joined with its position in the phrase to signal its role. Figure 1b shows an example in dependency parsing, where multiple types (words) are conjoined with POS tags or distance information. ∗Paper submitted during Mo Yu’s PhD study at HIT. To avoid model over-fitting that often results from features with lexical components, several smoothed lexical representations have been proposed and shown to improve performance on various NLP tasks; for instance, word embeddings (Bengio et al., 2006) help improve NER, dependency parsing and semantic role labeling (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010; Sun et al., 2011; Roth and Woodsend, 2014; Hermann et al., 2014). However, using only word embeddings is not suf- ficient to represent complex lexical features (e.g. φ in Figure 1c). In these features, the same word em- bedding conjoined with different non-lexical prop- erties may result in features indicating different la- bels; the corresponding lexical feature representa- tions should take the above interactions into consid- eration. Such important interactions also increase the risk of over-fitting as feature space grows ex- ponentially, yet how to capture these interactions in representation learning remains an open question. To address the above problems,1 we propose a general and unified approach to reduce the feature space by constructing low-dimensional feature rep- resentations, which provides a new way of combin- ing word embeddings, traditional non-lexical prop- erties, and label information. Our model exploits the inner structure of features by breaking the fea- ture into multiple parts: lexical, non-lexical and (op- tional) label. We demonstrate that the …

Content truncated to 500 words