Forming Trees with Treeformers

Proceedings of Recent Advances in Natural Language Processing, pages 836–845 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_090 836 Forming Trees with Treeformers Nilay Patel University of California, Santa Cruz nilay@ucsc.edu Jeffrey Flanigan University of California, Santa Cruz jmflanig@ucsc.edu Abstract Human language is known to exhibit a nested, hierarchical structure, allowing us to form com- plex sentences out of smaller pieces. However, many state-of-the-art neural networks models such as Transformers have no explicit hierar- chical structure in their architecture—that is, they don’t have an inductive bias toward hier- archical structure. Additionally, Transformers are known to perform poorly on compositional generalization tasks which require such struc- tures. In this paper, we introduce Treeformer, a general-purpose encoder module inspired by the CKY algorithm which learns a composition operator and pooling function to construct hi- erarchical encodings for phrases and sentences. Our extensive experiments demonstrate the ben- efits of incorporating hierarchical structure into the Transformer and show significant improve- ments in compositional generalization as well as in downstream tasks such as machine trans- lation, abstractive summarization, and various natural language understanding tasks. 1 Introduction Human language is known to exhibit a nested or hierarchical structure (Chomsky, 1956; Montague, 1970). This structure allows humans to construct complex sentences from simple parts and is im- portant for conveying meaning. For example, the phrase structure of the English sentence “The old man the boat.” is critical for correctly determining its meaning (Figure 1). Transformer models (Vaswani et al., 2017) are state-of-the-art across a wide variety of NLP tasks (Devlin et al., 2019), and pretrained Transformers have been shown to learn hierarchical structures af- ter pretraining on large amounts of data (Lin et al., 2019; Rogers et al., 2020). However, Transform- ers do not have a hierarchical structure built into the architecture—that is, they don’t have an induc- tive bias toward hierarchical structure (Tran et al., the old man the boat the old man the boat Figure 1: Two different parses of the text “the old man the boat” with significantly distinct meanings. While the top parse is a complete sentence (with “man” as a verb), the second is nonsense. Therefore, the encodings for the subphrase “the old man” (for example) in these parses should be significantly different. 2018). Additionally, Transformers are shown not to perform well on some compositional generaliza- tion tasks that require nested structure (Li et al., 2021). We demonstrate that incorporating an inductive bias toward the hierarchical structure of language improves the performance of the Transformer on downstream tasks. We show that this improves compositional generalization and greatly improves the translation of predicated argument structure in machine translation. Specifically, we augment the Transformer to make it more compositional by adding a tree-encoder layer designed for modeling hierarchical phrases. Additionally, we show this layer improves downstream performance across a wide variety of tasks. Our inductive bias layer, which we call Treeformer, is an encoder module that constructs hierarchical phrase encodings and is inspired by the CKY context-free-grammar parsing algorithm 837 (Cocke, 1969; Younger, 1966; Kasami, 1965). To …

Content truncated to 500 words