Evaluation of Domain-specific Word Embeddings using Knowledge Resources

Evaluation of Domain-specific Word Embeddings using Knowledge Resources Farhad Nooralahzadeh, Lilja Øvrelid, Jan Tore Lønning Department of Informatics, University of Oslo {farhadno,liljao,jtl}@ifi.uio.no Abstract In this work we evaluate domain-specific embedding models induced from textual resources in the Oil and Gas domain. We conduct intrinsic and extrinsic evaluations of both general and domain-specific embeddings and we observe that constructing domain-specific word embeddings is worthwhile even with a considerably smaller corpus size. Although the intrinsic evaluation shows low performance in synonymy detection, an in-depth error analysis reveals the ability of these models to discover additional semantic relations such as hyponymy, co-hyponymy and relatedness in the target domain. Extrinsic evaluation of the embedding models is provided by a domain-specific sentence classification task, which we solve using a convolutional neural network. We further adapt embedding enhancement methods to provide vector representations for infrequent and unseen terms. Experiments show that the adapted technique can provide improvements both in intrinsic and extrinsic evaluation. Keywords: word embeddings, intrinsic and extrinsic evaluation, domain knowledge resource, embeddings enhancement 1. Introduction Domain-specific, technical vocabulary presents a challenge to NLP applications. Recently, word embedding models have been shown to capture a range of semantic relations relevant to the interpretation of lexical items (Mikolov et al., 2013b) and furthermore provide useful input represen- tations for a range of downstream tasks (Collobert et al., 2011). The majority of work dealing with intrinsic evalu- ation of word embeddings has focused on general domain embeddings and semantic relations between frequent and generic terms. However, it has been shown that embed- dings differ from one domain to another due to lexical and semantic variation (Hamilton et al., 2016; Bollegala et al., 2015). Domain-specific terms are challenging for general domain embeddings since there are few statistical clues in the underlying corpora for these items (Bollegala et al., 2015; Pilehvar and Collier, 2016). The Oil and Gas domain is a highly technical and data- intensive domain. Experts working within this domain daily investigate selected geographical areas and use rel- evant information (scientific articles, reports and other tex- tual sources) to evaluate the potential for undiscovered hy- drocarbons. The vocabulary is technical and there is a real need for NLP tools to aid the work process. In this work we investigate whether word embedding models can cap- ture domain-specific semantic relations by training domain- specific embeddings1 and evaluating these against a termi- nological resource. We conduct a comprehensive study in- cluding a wide range of evaluation criteria, contrasting sev- eral general and domain specific embedding models. We augment the domain-specific embeddings using a domain knowledge resource. To supply embeddings for rare words, we extend the retrofitting method by Faruqui et al. (2015). We then go on to examine the contribution of these models in the performance of a downstream classification task. 1Link to the domain-specific model: http://vectors. nlpl.eu/repository/11/75.zip 2. Related work Despite the pervasive use of word embedding in language technology, there is no agreement in the community on the best ways to evaluate these semantic representations of lan- …

Content truncated to 500 words