Subword-based Compact Reconstruction of Word Embeddings

Proceedings of NAACL-HLT 2019, pages 3498–3508 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3498 Subword-based Compact Reconstruction of Word Embeddings Shota Sasaki1,2, Jun Suzuki2,1, Kentaro Inui2,1 1RIKEN AIP, 2Tohoku University {sasaki.shota, jun.suzuki, inui}@ecei.tohoku.ac.jp Abstract The idea of subword-based word embeddings has been proposed in the literature, mainly for solving the out-of-vocabulary (OOV) word problem observed in standard word-based word embeddings. In this paper, we propose a method of reconstructing pre-trained word embeddings using subword information that can effectively represent a large number of subword embeddings in a considerably small fixed space. The key techniques of our method are twofold: memory-shared embeddings and a variant of the key-value-query self-attention mechanism. Our experiments show that our reconstructed subword-based embeddings can successfully imitate well-trained word embed- dings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets, and can simultaneously predict effective embeddings of OOV words. We also demonstrate the effectiveness of our reconstruction method when we apply them to downstream tasks1. 1 Introduction Pre-trained word embeddings (or embedding vec- tors), especially those trained on a vast amount of text data, such as the Common Crawl (CC) cor- pus2, are now considered as highly beneficial, fun- damental language resources. Typical examples of large, well-trained word embeddings are those trained on the CC corpus with 600 billion tokens by fastText (Bojanowski et al., 2017) and with 840 billion tokens by GloVe (Pennington et al., 2014), which we refer to as fastText.600B3 and GloVe.840B4, respectively. In fact, we often 1Our code and reconstructed subword-based word embed- dings trained from GloVe.840B and fastText.600B are available: https://github.com/losyer/ compact_reconstruction 2http://commoncrawl.org 3https://fasttext.cc/docs/en/ english-vectors.html 4https://nlp.stanford.edu/projects/ glove/ leverage such word embeddings to further improve the task performance of many natural language processing (NLP) tasks, such as constituency parsing (Suzuki et al., 2018; G´omez-Rodr´ıguez and Vilares, 2018), discourse parsing (Yu et al., 2018), semantic parsing (Groschwitz et al., 2018; Dong and Lapata, 2018), and semantic role label- ing (Strubell et al., 2018). Despite their significant impact on the NLP community, well-trained word embeddings still have several disadvantages. In this paper, we focus on two issues surrounding well-trained word em- beddings: i) the massive memory requirement and ii) the inapplicability of out-of-vocabulary (OOV) words. It is crucial to address such issues, es- pecially when applying them to real-world open systems. The total number of embeddings (i.e., the total memory requirement of such word em- beddings) often becomes unacceptably large, es- pecially in limited-memory environments, includ- ing GPUs, since the vocabulary size is more than 2 million words, which require at least 2 gigabytes (GB) of memory for storage. One possible solution is to merely discard (less important) words from the vocabulary, which can straightforwardly reduce the memory requirement. However, such a naive method can cause another well-known drawback regarding the inapplicabil- ity of OOV words. The applicability of OOV words is highly desirable in real systems since in- put words can be uncontrollably diverse. There- fore, there is a trade-off between the number …

Content truncated to 500 words