Representation Learning for Unseen Words by Bridging Subwords to Semantic Networks

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4774–4780 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4774 Representation Learning for Unseen Words by Bridging Subwords to Semantic Networks Yeachan Kim1, Kang-Min Kim2, SangKeun Lee2 Artificial Intelligence Research Institute (AIRI)1, Korea University, Republic of Korea2 yeachan@airi.kr, {kangmin89, yalphy}@korea.ac.kr Abstract Pre-trained word embeddings are widely used in various fields. However, the coverage of pre-trained word embeddings only includes words that appeared in corpora where pre-trained embeddings are learned. It means that the words which do not appear in training corpus are ignored in tasks, and it could lead to the limited performance of neural models. In this paper, we propose a simple yet effective method to represent out-of-vocabulary (OOV) words. Unlike prior works that solely utilize subword information or knowledge, our method makes use of both information to represent OOV words. To this end, we propose two stages of representation learning. In the first stage, we learn subword embeddings from the pre-trained word embeddings by using an additive composition function of subwords. In the second stage, we map the learned subwords into semantic networks (e.g., WordNet). We then re-train the subword embeddings by using lexical entries on semantic lexicons that could include newly observed subwords. This two-stage learning makes the coverage of words broaden to a great extent. The experimental results clearly show that our method provides consistent performance improvements over strong baselines that use subwords or lexical resources separately. Keywords: Lexicon, Semantics, Knowledge Representation 1. Introduction Word embeddings have shown to be effective for improving many natural language processing (NLP) tasks, such as ma- chine translation, question answering. Recently, contextu- alized word embeddings such as BERT (Devlin et al., 2019) have largely improved the performance of NLP tasks com- pared to static embeddings such as word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014). However, static embeddings are still frequently used to various studies for sentence embeddings (Yang et al., 2019; Almarwani et al., 2019) and even the other domains such as the extraction of interaction between drugs in the biomedical field (Sun et al., 2019). In particular, pre-trained word embeddings, which are learned at large corpora, are frequently used and they are publicly available on the web. Despite of its usefulness and easy-accessibility, there is a major drawback to using pre-trained word embeddings. They are blind to out-of-vocabulary (OOV) words1 that are not included in the pre-defined vocabulary. This limitation could ignore significant words in tasks and lead to the lim- ited performance of neural models. Unfortunately, this is more evident in real-world scenarios since words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. There are two mainstreams to handle such OOV words. The first is to take the pre-trained word embeddings and gen- eralize them to all word entries using subwords (Pinter et al., 2017; Kim et al., 2018; Zhao et al., 2018). These ap- proaches effectively handle surface-variation …

Content truncated to 500 words