Documents Representation via Generalized Coupled Tensor Chain with the Rotation Group constraint

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1674–1684 August 1–6, 2021. ©2021 Association for Computational Linguistics 1674 Documents Representation via Generalized Coupled Tensor Chain with the Rotation Group Constraint Igor Vorona, Anh-Huy Phan, Alexander Panchenko and Andrzej Cichocki Skolkovo Institute of Science and Technology, Moscow, Russia {igor.vorona,a.phan, a.panchenko, a.cichocki}@skoltech.ru Abstract Continuous representations of linguistic struc- tures are an important part of modern natu- ral language processing systems. Despite the diversity, most of the existing log-multilinear embedding models are organized under vector operations. However, these operations can not precisely represent the compositionality of nat- ural language due to a lack of order-preserving properties. In this work, we focus on one of the promising alternatives based on the embed- ding of documents and words in the rotation group through the generalization of the cou- pled tensor chain decomposition to the expo- nential family of the probability distributions. In this model, documents and words are rep- resented as matrices, and n-grams representa- tions are combined from word representations by matrix multiplication. The proposed model is optimized via noise-contrastive estimation. We show empirically that capturing word or- der and higher-order word interactions allows our model to achieve the best results in several document classification benchmarks. 1 Introduction The current progress in natural language process- ing systems is largely based on the success of the representation learning of linguistic structures such as word, sentence and document embeddings. The most promising and successful methods are based on learning representations via two types of models: shallow log-multilinear models and deep neural networks. Despite the efficiency and interpretabil- ity of log-multilinear models, they can not use higher-order linguistic features like dependency between subsequences of words. To avoid these disadvantages, we usually use nonlinear predictors like recurrent, recursive, convolution neural net- works, and more recently Transformers. Neverthe- less, these methods can achieve high performance at the cost of loss of some interpretability and the cost of increased computation time. However, there exist other ways to utilize higher- order interactions and still preserve the efficiency of linear models. In this research, we focus on more data-oriented, i.e., more linguistic grounded, and better interpretable methods which still can achieve high results in the practical tasks. Particularly, we investigate the matrix-space model of language, in which semantic space consists of square matrices of real values. The key idea behind this method goes from realization of the Frege’s principle of compositionality through order-preserving prop- erty of matrix operations. As shown by Rudolph and Giesbrecht (2010), this type of models can in- ternally combine various properties from statistical and symbolic models of natural language and there- fore it is more flexible than vector space models. In spite of that fact, such models are usually hard to optimize on the real data. To this end, Yesse- nalina and Cardie (2011) took attention to the needs of nontrivial initialization and proposed to learn the weights by the bag-of-words model. Asaadi and Rudolph (2017) used complex multi-stage initial- ization based on unigrams and bigrams …

Content truncated to 500 words