Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 1204–1213 Barcelona, Spain (Online), December 8-13, 2020 1204 Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models Phong Ha1 Shanshan Zhang1 Nemanja Djuric2 Slobodan Vucetic1 1Department of Computer and Information Sciences, Temple University 2Uber ATG {phongtheha, zhang.shanshan, nemanja, vucetic}@temple.edu Abstract Embedding of rare and out-of-vocabulary (OOV) words is an important open NLP problem. A popular solution is to train a character-level neural network to reproduce the embeddings from a standard word embedding model. The trained network is then used to assign vectors to any input string, including OOV and rare words. We enhance this approach and introduce an algorithm that iteratively refines and improves both word- and character-level models. We demonstrate that our method outperforms the existing algorithms on 5 word similarity data sets, and that it can be successfully applied to job title normalization, an important problem in the e-recruitment domain that suffers from the OOV problem. 1 Introduction Inability to represent rare and unseen words, which is referred to as the out-of-vocabulary (OOV) problem, limits usefulness of the standard embedding approaches to real-world applications. For example, in the e-recruitment domain there is an extremely large number of ways one can write a job title for the same job type. LinkedIn users who are software engineers may describe themselves as software developer, python guru, sw engi., sw developper or sw and web application developer, which are caused by different variants, abbreviations, misspellings, or compoundings. In this case, standard embedding for a finite set of job titles is not helpful when recommending jobs to users with rare or unseen job titles. An embedding model that can effectively handle the OOV problem should preserve both semantic and syntactic characteristics of words and phrases. Recently proposed mimicking approach is a promising solution to this problem, with two representative algorithms being Mimick (Pinter et al., 2017) and GWR (Kim et al., 2018). Their main idea is to train a character-level embedding neural network (NN) that can reconstruct, or mimic, an embedding from word-level embedding model. The trained character-level model is then used to generate both semantically and syntactically relevant embeddings for arbitrary character sequences. While Mimick and GWR used BiLSTM and CNN character-level models, respectively, similar approaches were recently proposed with several other architectures (Zhao et al., 2018; Schick and Schütze, 2019). In order to provide high-quality embeddings useful for downstream applications, it is crucial for the model to preserve semantic characteristics of words and phrases, while also being robust to syntactic and word-form changes. However, this may not always be the case for the vanilla mimicking method. To see why, let us first denote the word embedding model as W and the character embedding model as G in the mimicking mechanism. Let us take a peek at model W produced by training on career profile corpus, described in detail in Section 5. The top 5 neighbors for word java developer in the vector space induced by W are software developer, …

Content truncated to 500 words