Learning from Non-Binary Constituency Trees via Tensor Decomposition

Proceedings of the 28th International Conference on Computational Linguistics, pages 3899–3910 Barcelona, Spain (Online), December 8-13, 2020 3899 Learning from Non-Binary Constituency Trees via Tensor Decomposition Daniele Castellana University of Pisa, Pisa daniele.castellana@di.unipi.it Davide Bacciu University of Pisa, Pisa bacciu@di.unipi.it Abstract Processing sentence constituency trees in binarised form is a common and popular approach in literature. However, constituency trees are non-binary by nature. The binarisation procedure changes deeply the structure, furthering constituents that instead are close. In this work, we in- troduce a new approach to deal with non-binary constituency trees which leverages tensor-based models. In particular, we show how a powerful composition function based on the canonical tensor decomposition can exploit such a rich structure. A key point of our approach is the weight sharing constraint imposed on the factor matrices, which allows limiting the number of model pa- rameters. Finally, we introduce a Tree-LSTM model which takes advantage of this composition function and we experimentally assess its performance on different NLP tasks. 1 Introduction One of the fundamental problems in Natural Language Processing (NLP) is learning a distributed en- coding of sentences, as this is the stepping stone for many NLP tasks, such as sentence classification, sentiment analysis and natural language inference. The multitude of approaches addressing this problem can be categorised according to how a sentence is represented. The simpler sentence representation is bag-of-words, which depicts sentences as words multisets ig- noring the word order. Despite the simple representation, it has been used to obtain meaningful sentence encodings (Iyyer et al., 2015; Wieting et al., 2016; Arora et al., 2017). Sequence representation overcomes this limitation considering the sentence as an ordered sequence of words. It allows building models which progressively constructs a sentence encoding, processing one word at the time. Recurrent Neural Network (Elman, 1990) and Long-Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) are probably the most famous models which use this representation. A key aspect of sentences, which is missing in sequential processing, is compositionality. For example, the sentence ”The sky is blue and the grass is green” is obtained by composing the two sub-phrases ”The sky is blue” and ”the grass is green” with the conjunction ”and”. The intrinsic compositionality of sentences makes them suitable for a tree representation, where the whole sentence (the root) is built in terms of sub-phrases (the internal nodes) which in turn are defined in terms of smaller constituents; the base cases are words (the leaves) since they are the atomic piece of information. This representation takes the name of constituency tree. In Fig. 1a we show the constituency tree of the sentence ”Effective but too-tepid biopic”: the leaves are the words while internal nodes represent syntactic categories which are the constituents of the whole sentence. There are many models which compute a sentence encoding starting from its constituency tree. For our purposes, we restrict the discussion on bottom-up Recursive Neural Networks (RecNNs) (Goller and Kuchler, 1996; Frasconi et al., 1998). The parsing direction is constrained by the structure …

Content truncated to 500 words