Representation Learning for Type-Driven Composition

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 313–324 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 313 Representation Learning for Type-Driven Composition Gijs Wijnholds Utrecht Institute of Linguistics OTS Utrecht University g.j.wijnholds@uu.nl Mehrnoosh Sadrzadeh Department of Computer Science University College London m.sadrzadeh@ucl.ac.uk Stephen Clark School of Electronic Engineering and Computer Science Queen Mary University of London stephen.clark609@gmail.com Abstract This paper is about learning word representa- tions using grammatical type information. We use the syntactic types of Combinatory Catego- rial Grammar to develop multilinear represen- tations, i.e. maps with n arguments, for words with different functional types. The multilin- ear maps of words compose with each other to form sentence representations. We extend the skipgram algorithm from vectors to multi- linear maps to learn these representations and instantiate it on unary and binary maps for tran- sitive verbs. These are evaluated on verb and sentence similarity and disambiguation tasks and a subset of the SICK relatedness dataset. Our model performs better than previous type- driven models and is competitive with state of the art representation learning methods such as BERT and neural sentence encoders. 1 Introduction We develop a novel technique for learning word representations by using syntactic type information of the words to learn representations for them and the constituency-based structure of the sentence to compose the representations. The word representa- tions are multilinear maps, i.e. maps with variable number of arguments, where the number of argu- ments and the type of each map come from the syntactic type of each word. The word representa- tion are composed via the application and further composition of the results of these maps, based on constituency structure. For instance, a noun such as children or ball is represented by a vector, i.e. −−−−−→ children or −−→ ball, which can be thought of as 0-linear maps as they have no input or output. An adjective such as young is represented by a unilinear map young, i.e. a lin- ear map of one argument, which at input takes an argument of type noun, e.g. children and at output returns an argument of type noun, i.e. young chil- dren. A transitive verb such as play, is represented by a bilinear map play, i.e. a linear map with two arguments, which at input takes two arguments of type noun, e.g. children and ball, and at output returns an argument of type sentence, i.e. young children play ball. An adjective-noun phrase repre- sentation is obtained by applying the representation of the adjective to the representation of its noun, i.e. by applying the unilinear map of the adjective to the vector of the noun. A sentence representation is obtained by the composition of two applications, i.e. by first applying the representation of the verb, e.g. play to the representation of the object, e.g. −−→ ball, resulting in a unilinear map for the representa- tion of the verb phrase play ball, and subsequently applying this verb phrase to the representation of the subject, e.g. …

Content truncated to 500 words