Right-truncatable Neural Word Embeddings

Proceedings of NAACL-HLT 2016, pages 1145–1151, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Right-truncatable Neural Word Embeddings Jun Suzuki and Masaaki Nagata NTT Communication Science Laboratories, NTT Corporation 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237 Japan {suzuki.jun, nagata.masaaki}@lab.ntt.co.jp Abstract This paper proposes an incremental learning strategy for neural word embedding methods, such as SkipGrams and Global Vectors. Since our method iteratively generates embedding vectors one dimension at a time, obtained vec- tors equip a unique property. Namely, any right-truncated vector matches the solution of the corresponding lower-dimensional embed- ding. Therefore, a single embedding vector can manage a wide range of dimensional re- quirements imposed by many different uses and applications. 1 Introduction Word embedding vectors obtained from ‘neural word embedding methods’, such as SkipGram, continuous bag-of-words (CBoW) and the fam- ily of vector log-bilinear (vLBL) models (Mnih and Kavukcuoglu, 2013; Mikolov et al., 2013a; Mikolov et al., 2013c; Mikolov et al., 2013b) have now become an important fundamental resource for tackling many natural language processing (NLP) tasks. These NLP tasks include part-of-speech tag- ging (Tsuboi, 2014; Ling et al., 2015), dependency parsing (Chen and Manning, 2014; Dyer et al., 2015; Alberti et al., 2015), semantic role labeling (Zhou and Xu, 2015; Woodsend and Lapata, 2015), ma- chine translation (Sutskever et al., 2014), sentiment analysis (Kim et al., 2015), and question answer- ing (Wang and Nyberg, 2015). The main purpose of this paper is to further en- hance the ‘usability’ of obtained embedding vectors in actual use. To briefly explain our motivation, we first introduce the following concept: Definition 1 (D′-right-truncated vector1). Let w′ and w′′ be vectors, whose dimensions are D′ and D′′, respectively. Namely, w′ = (w′ 1, . . . , w′ D′) and w′′ = (w′′ 1, . . . , w′′ D′′). Suppose w matches the concatenation of w′ and w′′, that is, w = (w′ 1, . . . , w′ D′, w′′ 1, . . . , w′′ D′′). Then, we define w′ as a D′-right-truncated vector of w. This paper focuses on the fact that the appropriate dimension of embedding vectors strongly depends on applications and uses, and is basically determined based on the performance and memory space (or calculation speed) trade-off. Indeed, the actual di- mensions of the previous studies listed above are di- verse; often around 50, and at most 1000. It is worth noting here that each dimension of embedding vec- tors obtained by conventional methods has no inter- pretable meaning. Thus, we basically need to re- train D′-dimensional embedding vectors even if we already have a well-trained D-dimensional vector. In addition, we cannot take full advantage of freely available high-quality pre-trained embedding vec- tors2 since their dimensions are already given and fixed, i.e., D=300. To reduce the additional computational cost of the retraining, and to improve the ‘usability’ of embed- ding vectors, we propose a framework for incremen- tally determining embeddings one dimension at a time from 1 to D. As a result, our method always …

Content truncated to 500 words