CBOW-tag: a Modified CBOW Algorithm for Generating Embedding Models from Annotated Corpora

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4798–4801 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4798 CBOW-tag: a Modified CBOW Algorithm for Generating Embedding Models from Annotated Corpora Attila Novák, László János Laki, Borbála Novák MTA-PPKE Hungarian Language Technology Research Group, Pázmány Péter Catholic University, Faculty of Information Technology and Bionics Práter u. 50/a, 1083 Budapest, Hungary {surname.firstname}@itk.ppke.hu Abstract In this paper, we present a modified version of the CBOW algorithm implemented in the fastText framework. Our modified algorithm, CBOW-tag builds a vector space model that includes the representation of the original word forms and their annotation at the same time. We illustrate the results by presenting a model built from a corpus that includes morphological and syntactic annotations. The simultaneous presence of unannotated elements and different annotations at the same time in the model makes it possible to constrain nearest neighbour queries to specific types of elements. The model can thus efficiently answer questions such as What do we eat?, What can we do with a skeleton? What else do we do with what we eat?, etc. Error analysis reveals that the model can highlight errors introduced into the annotation by the tagger and parser we used to generate the annotations as well as lexical peculiarities in the corpus itself, especially if we do not limit the vocabulary of the model to frequent items. Keywords: word and annotation embedding, annotation, lexical resources 1. Introduction An efficient implementation of two simple feed-forward neural network architectures, CBOW (continous bag-of- words) and skip-gram in the word2vec1 tool (Mikolov et al., 2013a; Mikolov et al., 2013b), has revolutionized the cre- ation of distributional semantic models. The networks are trained to predict a target word from the context (CBOW) or vice versa (skip-gram). The embedding vectors can be extracted from the middle layer of the network and can be used alike in both cases. The fastText2 algorithm (Bojanowski et al., 2017) extends the implementation of word2vec by creating vector rep- resentations for character n-grams in addition to words. When training this model, the context of a target word (and of its n-gram fragments) is considered to consist of not only other words but also n-gram fragments of those words. However, all these implementations are trained on a corpus that is a simple sequence of tokens. It is possible to replace words in the token sequence with some sort of annotation, e.g. lemmas, or lemmas followed by morphological anno- tation (Novák and Novák, 2018), but in that case the repre- sentation of word forms will not be part of the model. Al- though models built from a corpus including grammatical annotation perform better in certain tasks than models built from raw corpora (Ebert et al., 2016; Novák and Novák, 2018), in most NLP tasks, the vector representation of the surface forms of words is also needed. An alternative ap- proach is presented by Levy and Goldberg (2014), where the skipgram model is modified …

Content truncated to 500 words