Improving Constituent Representation with Hypertree Neural Networks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1682 - 1692 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving Constituent Representation with Hypertree Neural Networks Hao Zhou3 , Gongshen Liu3 , Kewei Tu†∗ 3 School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University † School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging {zhou1998,lgshen}@sjtu.edu.cn tukw@shanghaitech.edu.cn Abstract Many natural language processing tasks in- volve text spans and thus high-quality span representations are needed to enhance neural approaches to these tasks. Most existing meth- ods of span representation are based on sim- ple derivations (such as max-pooling) from word representations and do not utilize compo- sitional structures of natural language. In this paper, we aim to improve representations of constituent spans using a novel hypertree neural networks (HTNN) that is structured with con- stituency parse trees. Each node in the HTNN represents a constituent of the input sentence and each hyperedge represents a composition of smaller child constituents into a larger par- ent constituent. In each update iteration of the HTNN, the representation of each constituent is computed based on all the hyperedges con- nected to it, thus incorporating both bottom-up and top-down compositional information. We conduct comprehensive experiments to evalu- ate HTNNs against other span representation models and the results show the effectiveness of HTNN. 1 Introduction Distributed span representations are useful in vari- ous natural language processing tasks such as ques- tion answering (Seo et al., 2019), coreference res- olution (Lee et al., 2017), sentiment classification (Yin et al., 2020) and semantic role labeling (He et al., 2018). Since spans can have arbitrary length, to get fix-dimensional span representations, exist- ing methods are mostly based on simple derivations from word or sub-word representations. These methods either apply some form of pooling over all the words within the target span or utilize only the boundary words of the target span to compute the span representation. There have also been efforts ∗Kewei Tu is the corresponding author. This work was conducted when Hao Zhou was a visiting student at Shang- haiTech University. Figure 1: An example binarized constituency parse tree (a) and its corresponding hypertree (b). For each hy- peredge, P/L/R denote the parent, left-child, right-child constituent spans respectively. to improve word representation learning using in- formation from text spans, which may at the same time improve span representations built on top of word representations (Joshi et al., 2020). Further, some previous methods introduce short text spans (phrases or n-grams) as additional inputs to BERT and thus learn contextual representations for such spans along with word representations (Lai et al., 2021; Zhang et al., 2021). However, it is known that natural language text has underlying compositional structure, which can often be represented with constituency parse trees (Chomsky, 1957). What is missing from the afore- mentioned span representation methods is the uti- lization of compositional structures both within and outside …

Content truncated to 500 words