BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages Benjamin Heinzerling and Michael Strube AIPHES, Heidelberg Institute for Theoretical Studies Schloss-Wolfsbrunnenweg 35, 69118 Heidelberg, Germany {benjamin.heinzerling | michael.strube}@h-its.org Abstract We present BPEmb, a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages bet- ter than alternative subword approaches, while requiring vastly fewer resources and no tokenization. BPEmb is available at https://github.com/bheinzerling/bpemb. Keywords: subword embeddings, byte-pair encoding, multilingual 1. Introduction Learning good representations of rare words or words not seen during training at all is a difficult challenge in natu- ral language processing. As a makeshift solution, systems have typically replaced such words with a generic UNK to- ken. Recently, based on the assumption that a word’s mean- ing can be reconstructed from its parts, several subword- based methods have been proposed to deal with the un- known word problem: character-based recurrent neural net- works (RNN) (Luong and Manning, 2016), character-based convolutional neural networks (CNN) (Chiu and Nichols, 2016), word embeddings enriched with subword informa- tion (FastText) (Bojanowski et al., 2017), and byte-pair encoding (BPE) (Sennrich et al., 2016), among others. While pre-trained FastText embeddings are publicly avail- able, embeddings for BPE units are commonly trained on a per-task basis (e.g. a specific language pair for machine- translation) and not published for general use. In this work we present BPEmb, a collection of pre-trained subword embeddings in 275 languages, and make the fol- lowing contributions: • We publish BPEmb, a collection of pre-trained byte- pair embeddings in 275 languages; • We show the utility of BPEmb in a fine-grained entity typing task; and • We show that BPEmb performs as well as, and for some languages better than, alternative approaches while being more compact and requiring no tokeniza- tion. 2. BPEmb: Byte-pair Embeddings Byte Pair Encoding is a variable-length encoding that views text as a sequence of symbols and iteratively merges the most frequent symbol pair into a new symbol. E.g., encod- ing an English text might consist of first merging the most frequent symbol pair t h into a new symbol th, then merg- ing the pair th e into the in the next iteration, and so on. The number of merge operations o determines if the result- ing encoding mostly creates short character sequences (e.g. o = 1000) or if it includes symbols for many frequently occurring words, e.g. o = 30, 000 (cf. Table 1). Since the BPE algorithm works with any sequence of symbols, it re- quires no preprocessing and can be applied to untokenized text. We apply BPE1 to all Wikipedias2 of sufficient size with various o and pre-train embeddings for the resulting BPE symbol using GloVe (Pennington et al., 2014), resulting in byte-pair embeddings for 275 languages. To allow study- ing the effect the number of BPE merge operations and of the embedding dimensionality, we provide embeddings for 1000, 3000, 5000, 10000, 25000, 50000, 100000 …

Content truncated to 500 words