Combining Word Embeddings and Feature Embeddings for Fine-grained Relation Extraction

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1374–1379, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Combining Word Embeddings and Feature Embeddings for Fine-grained Relation Extraction Mo Yu ∗ Machine Intelligence & Translation Lab Harbin Institute of Technology Harbin, China gflfof@gmail.com Matthew R. Gormley, Mark Dredze Human Language Technology Center of Excellence Center for Language and Speech Processing Johns Hopkins University Baltimore, MD, 21218 {mgormley, mdredze}@cs.jhu.edu Abstract Compositional embedding models build a rep- resentation for a linguistic structure based on its component word embeddings. While re- cent work has combined these word embed- dings with hand crafted features for improved performance, it was restricted to a small num- ber of features due to model complexity, thus limiting its applicability. We propose a new model that conjoins features and word em- beddings while maintaing a small number of parameters by learning feature embeddings jointly with the parameters of a compositional model. The result is a method that can scale to more features and more labels, while avoiding overfitting. We demonstrate that our model at- tains state-of-the-art results on ACE and ERE fine-grained relation extraction. 1 Introduction Word embeddings represent words in some low- dimensional space, where each dimension might in- tuitively correspond to some syntactic or semantic property of the word.1 These embeddings can be used to create novel features (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010; Sun et al., 2011; Nguyen and Grishman, 2014; Roth and Woodsend, 2014), and can also be treated as model parameters ∗ The work was done while the author was visiting JHU. 1Such embeddings have a long history in NLP, such as term co-occurrence frequency matrices and their low-dimensional counterparts obtained by linear algebra tools (LSA, PCA, CCA, NNMF) and word clusters. Recently, neural networks have be- come popular methods for obtaining such embeddings (Bengio et al., 2006; Collobert et al., 2011; Mikolov et al., 2013). to build representations for higher-level structures in some compositional embedding models (Collobert et al., 2011; Collobert, 2011; Socher et al., 2012; Socher et al., 2013b; Hermann et al., 2014). Appli- cations of embedding have boosted the performance of many NLP tasks, including syntax (Turian et al., 2010; Collobert et al., 2011), semantics (Socher et al., 2012; Socher et al., 2013b; Hermann et al., 2014), question answering (Bordes et al., 2014) and machine translation (Devlin et al., 2014). While compositional models aim to learn higher- level structure representations, composition of em- beddings alone may not capture important syntac- tic or semantic patterns. Consider the task of re- lation extraction, where decisions require examin- ing long-distance dependencies in a sentence. For the sentence in Figure 1, “driving” is a strong in- dicator of the “ART” (ACE) relation because it ap- pears on the dependency path between a person and a vehicle. Yet such conjunctions of different syntac- tic/semantic annotations (dependency and NER) are typically not available in compositional models. In contrast, hand-crafted …

Content truncated to 500 words