ETNLP: A Visual-Aided Systematic Approach to Select Pre-Trained Embeddings for a Downstream Task

Proceedings of Recent Advances in Natural Language Processing, pages 1285–1294, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_147 1285 ETNLP: A Visual-Aided Systematic Approach to Select Pre-Trained Embeddings for a Downstream Task Xuan-Son Vu1, Thanh Vu2, Son N. Tran3, Lili Jiang1 1Ume˚a University, Sweden 2The Australian E-Health Research Centre, CSIRO, Australia 3 The University of Tasmania, Australia; {sonvx, lili.jiang}@cs.umu.se thanh.vu@csiro.au, sn.tran@utas.edu.au ; Abstract Given many recent advanced embedding mod- els, selecting pre-trained word embedding (a.k.a., word representation) models best fit for a specific downstream task is non-trivial. In this paper, we propose a systematic approach, called ETNLP, for extracting, evaluating, and visualizing multiple sets of pre-trained word embeddings to determine which embeddings should be used in a downstream task. We demonstrate the effectiveness of the pro- posed approach on our pre-trained word em- bedding models in Vietnamese to select which models are suitable for a named entity recogni- tion (NER) task. Specifically, we create a large Vietnamese word analogy list to evaluate and select the pre-trained embedding models for the task. We then utilize the selected embed- dings for the NER task and achieve the new state-of-the-art results on the task benchmark dataset. We also apply the approach to another downstream task of privacy-guaranteed em- bedding selection, and show that it helps users quickly select the most suitable embeddings. In addition, we create an open-source system using the proposed systematic approach to fa- cilitate similar studies on other NLP tasks. The source code and data are available at https: //github.com/vietnlp/etnlp. 1 Introduction Word embedding, also known as word represen- tation, represents a word as a vector capturing both syntactic and semantic information, so that the words with similar meanings should have similar vectors (Levy and Goldberg, 2014). Al- though, the classical embedding models, such as Word2Vec (Mikolov et al., 2013), GloVe (Pen- nington et al., 2014), fastText (Bojanowski et al., 2017), have been shown to help improve the per- formance of existing models in a variety of Nat- ural Language Processing (NLP) tasks like pars- ing (Bansal et al., 2014), topic modeling (Nguyen et al., 2015), and document classification (Taddy, 2015; Vu et al., 2018b). Each word is associated with a single vector leading to a challenge on us- ing the vector across linguistic contexts (Peters et al., 2018). To handle the problem, recently, con- textual embeddings (e.g., ELMO of Peters et al. (2018), BERT of Devlin et al. (2018)) have been proposed and help existing models achieve new state-of-the-art results on many NLP tasks. Dif- ferent from non-contextual embeddings, ELMO and BERT can capture different latent syntactic- semantic information of the same word based on its contextual uses. Therefore, for completeness, in this paper, we incorporate both classical em- beddings (i.e., Word2Vec, fastText) and contextual embeddings (i.e., ELMO, BERT) to evaluate their performances on NLP downstream tasks. Given the fact that there are many different types of word embedding models, we argue that having a systematic pipeline to evaluate, extract, and visualize word embeddings for a downstream NLP task, is important …

Content truncated to 500 words