PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 596–611 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 596 PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding Zhao Jinman Shawn Zhong Xiaomin Zhang Yingyu Liang University of Wisconsin-Madison, Madison, WI, USA {jz,yliang}@cs.wisc.edu {shawn.zhong,xzhang682}@wisc.edu Abstract We look into the task of generalizing word em- beddings: given a set of pre-trained word vec- tors over a finite vocabulary, the goal is to pre- dict embedding vectors for out-of-vocabulary words, without extra contextual information. We rely solely on the spellings of words and propose a model, along with an efficient al- gorithm, that simultaneously models subword segmentation and computes subword-based compositional word embedding. We call the model probabilistic bag-of-subwords (PBoS), as it applies bag-of-subwords for all possi- ble segmentations based on their likelihood. Inspections and affix prediction experiment show that PBoS is able to produce meaning- ful subword segmentations and subword rank- ings without any source of explicit morpho- logical knowledge. Word similarity and POS tagging experiments show clear advantages of PBoS over previous subword-level models in the quality of generated word embeddings across languages. 1 Introduction Word embeddings pre-trained over large texts have demonstrated benefits for many NLP tasks, espe- cially when the task is label-deprived. However, many popular pre-trained sets of word embeddings assume fixed finite-size vocabularies 1, 2, which hinders their ability to provide useful word repre- sentations for out-of-vocabulary (OOV) words. We look into the task of generalizing word em- beddings: extrapolating a set of pre-trained word embeddings to words out of its fixed vocabulary, without extra access to contextual information (e.g. example sentences or text corpus). In contrast, 1https://code.google.com/archive/p/ word2vec/, Mikolov et al. (2013). 2https://nlp.stanford.edu/projects/ glove/, Pennington et al. (2014). the more common task of learning word embed- dings, or often just word embedding, is to obtain distributed representations of words directly from large unlabeled text. The motivation here is to extend the usefulness of pre-trained embeddings without expensive retraining over large text. There have been works showing that contextual information can also help generalize word embed- dings (for example, Khodak et al., 2018; Schick and Sch¨utze, 2019a,b). We here, however, focus more on the research question of how much one can achieve from just word compositions. In addition, our proposed way of utilizing word composition information can be combined with the contextual embedding algorithms to further improve the per- formance of generalized embeddings. The hidden assumption here is that words are made of meaningful parts (cf. morphemes) and that the meaning of a word is related to the meaning of their parts. This way, humans are often able to guess the meaning of a word or term they have never seen before. For example, “postEMNLP” probably means “after EMNLP”. Different models have been proposed for that task of generalizing word embeddings using word compositions, usually under the name of subword(- level) models. Stratos (2017); Pinter et al. (2017); Kim et al. (2018b) model words at the charac- ter level. However, they have been surpassed by …

Content truncated to 500 words