Tiny Word Embeddings Using Globally Informed Reconstruction

Proceedings of the 28th International Conference on Computational Linguistics, pages 1199–1203 Barcelona, Spain (Online), December 8-13, 2020 1199 Tiny Word Embeddings Using Globally Informed Reconstruction Sora Ohashi†, Mao Isogawa†, Tomoyuki Kajiwara‡, Yuki Arase† † Graduate School of Information Science and Technology, Osaka University ‡ Institute for Datability Science, Osaka University † {ohashi.sora, isogawa.mao, arase}@ist.osaka-u.ac.jp ‡ kajiwara@ids.osaka-u.ac.jp Abstract We reduce the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. Previous studies in this direction created a smaller word embedding model by recon- structing pre-trained word representations from those of subwords, which allows to store only a smaller number of subword embeddings in the memory. However, previous studies that train the reconstruction models using only target words cannot reduce the model size extremely while preserving its quality. Inspired by the observation of words with similar meanings having simi- lar embeddings, our reconstruction training learns the global relationships among words, which can be employed in various models for word embedding reconstruction. Experimental results on word similarity benchmarks show that the proposed method improves the performance of the all subword-based reconstruction models. 1 Introduction Word embeddings form the basis for many natural language processing (NLP) applications, e.g., text classification (Shen et al., 2018) and machine translation (Qi et al., 2018). However, widely used pre- trained word embeddings such as fastText (Bojanowski et al., 2017) are considerably large, thereby making it difficult to develop NLP applications in limited memory environments such as mobile devices. For example, fastText1 (crawl-300d-2M-subword) requires approximately 2 GB of memory. In previous studies, the model size has been reduced by reconstructing word embeddings from charac- ters (Pinter et al., 2017; Kim et al., 2018) and character N-grams (Zhao et al., 2018; Sasaki et al., 2019). As the number of characters or character N-grams, is significantly smaller than that of words, reconstruct- ing word embeddings with accuracy from these subwords can reduce the model size while preserving the performance of applications.2 As shown in Figure 1, existing methods reconstruct word embeddings from subword embeddings and mimic the corresponding pre-trained word embeddings. These methods rely only on local information of subwords and pre-trained word embeddings. To improve the performance of word embedding reconstruction, we propose a global loss function that uses words other than the target word as clues. Inspired by the observation of words with similar mean- ings having similar embeddings in pre-trained word embeddings, our reconstruction training learns the similarity among word embeddings. Our method can be easily applied to any method for reconstructing a word embedding from subwords, regardless of the unit of the subword or the network structure. Experimental results on word similarity tasks (Faruqui and Dyer, 2014) show that our global loss function improves the performance of word embedding reconstruction in all previous methods. When the proposed method was applied to the method based on a self-attention mechanism for reconstructing word embeddings from character N-grams (Sasaki et al., 2019), the model size was reduced to 74 MB (1/30) while preserving 97% …

Content truncated to 500 words