Advances in Pre-Training Distributed Word Representations

Advances in Pre-Training Distributed Word Representations Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin Facebook AI Research {tmikolov, egrave, bojanowski, cpuhrsch, ajoulin}@fb.com Abstract Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks. Keywords: fastText, word2vec, word vectors, pre-trained 1. Introduction Pre-trained continuous word representations have become basic building blocks of many Natural Language Process- ing (NLP) and Machine Learning applications. These pre-trained representations provide distributional infor- mation about words, that typically improve the gen- eralization of models learned on limited amount of data (Collobert et al., 2011). This information is typically derived from statistics gathered from large unlabeled cor- pus of text data (Deerwester et al., 1990). A critical aspect of their training is thus to capture efficiently as much statis- tical information as possible from rich and vast sources of data. A standard approach for learning word representations is to train log-bilinear models based on either the skip-gram or the continuous bag-of-words (cbow) architectures, as implemented in word2vec (Mikolov et al., 2013a) and fast- Text (Bojanowski et al., 2017)1. In the skip-gram model, nearby words are predicted given a source word, while in the cbow model, the source word is predicted according to its context. These architectures and their implementation have been optimized to produce high quality word repre- sentations able to transfer to many tasks, while maintaining a sufficiently high training speed to scale to massive amount of data. Recently, word2vec representations have been widely used in NLP pipelines to improve their performance. Their im- pressive capability at transfering to new problems suggests that they are capturing important statistics about the train- ing corpora (Baroni and Lenci, 2010). As can be expected, the more data a model is trained on, the better the represen- tations are at transferring to other NLP problems. Training such models on massive data sources, like Common Crawl, can be cumbersome and many NLP practitioners prefer to use publicly available pre-trained word vectors over train- ing the models by themselves. In this work, we provide new pre-trained word vectors that show consistent improvement over the currently available ones, making them potentially very useful to a wide community of researchers. We show that several modifications of the standard word2vec training pipeline significantly improves the qual- ity of the resulting word vectors. We focus mainly on 1https://fasttext.cc/ known modifications and data pre-processing strategies that are rarely used together: the position dependent features in- troduced by Mnih and Kavukcuoglu (2013), the phrase rep- resentations used in Mikolov et al. (2013b) and the use of subword information (Bojanowski et al., 2017). We measure their …

Content truncated to 500 words