Tree Transformer: Integrating Tree Structures into Self-Attention

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1061–1070, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1061 Tree Transformer: Integrating Tree Structures into Self-Attention Yau-Shian Wang Hung-Yi Lee Yun-Nung Chen National Taiwan University, Taipei, Taiwan king6101@gmail.com hungyilee@ntu.edu.tw y.v.chen@ieee.org Abstract Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks. However, it is unclear what the learned attention captures. The attention com- puted by attention heads seems not to match human intuitions about hierarchical structures. This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures. The tree structures can be automat- ically induced from raw texts by our proposed “Constituent Attention” module, which is sim- ply implemented by self-attention between two adjacent words. With the same training procedure identical to BERT, the experiments demonstrate the effectiveness of Tree Trans- former in terms of inducing tree structures, better language modeling, and further learning more explainable attention scores1. 1 Introduction Human languages exhibit a rich hierarchical struc- ture which is currently not exploited nor mir- rored by the self-attention mechanism that is the core of the now popular Transformer archi- tecture. Prior work that integrated hierarchi- cal structure into neural networks either used re- cursive neural networks (Tree-RNNs) (C.Goller and A.Kuchler, 1996; Socher et al., 2011; Tai et al., 2015) or simultaneously generated a syn- tax tree and language in RNN (Dyer et al., 2016), which have shown beneficial for many downstream tasks (Aharoni and Goldberg, 2017; Eriguchi et al., 2017; Strubell et al., 2018; Zare- moodi and Haffari, 2018). Considering the re- quirement of the annotated parse trees and the 1The source code is publicly available at https:// github.com/yaushian/Tree-Transformer. costly annotation effort, most prior work relied on the supervised syntactic parser. However, a su- pervised parser may be unavailable when the lan- guage is low-resourced or the target data has dif- ferent distribution from the source domain. Therefore, the task of learning latent tree struc- tures without human-annotated data, called gram- mar induction (Carroll and Charniak, 1992; Klein and Manning, 2002; Smith and Eisner, 2005), has become an important problem and attractd more attention from researchers recently. Prior work mainly focused on inducing tree structures from recurrent neural networks (Shen et al., 2018a,b) or recursive neural networks (Yogatama et al., 2017; Drozdov et al., 2019), while integrating tree struc- tures into Transformer remains an unexplored di- rection. Pre-training Transformer from large-scale raw texts successfully learns high-quality language representations. By further fine-tuning pre-trained Transformer on desired tasks, wide range of NLP tasks obtain the state-of-the-art results (Radford et al., 2019; Devlin et al., 2018; Dong et al., 2019). However, what pre-trained Transformer self-attention heads capture remains unknown. Al- though an attention can be easily explained by ob- serving how words attend to each other, only …

Content truncated to 500 words