Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1336–1341, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1336 Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons Jie Hao∗ Florida State University haoj8711@gmail.com Xing Wang Tencent AI Lab brightxwang@tencent.com Shuming Shi Tencent AI Lab shumingshi@tencent.com Jinfeng Zhang Florida State University jinfeng@stat.fsu.edu Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Recent studies have shown that a hybrid of self-attention networks (SANs) and recur- rent neural networks (RNNs) outperforms both individual architectures, while not much is known about why the hybrid models work. With the belief that modeling hierarchical structure is an essential complementary be- tween SANs and RNNs, we propose to further enhance the strength of hybrid models with an advanced variant of RNNs – Ordered Neurons LSTM (ON-LSTM, Shen et al., 2019), which introduces a syntax-oriented inductive bias to perform tree-like composition. Experimental results on the benchmark machine translation task show that the proposed approach outper- forms both individual architectures and a stan- dard hybrid model. Further analyses on tar- geted linguistic evaluation and logical infer- ence tasks demonstrate that the proposed ap- proach indeed benefits from a better modeling of hierarchical structure. 1 Introduction Self-attention networks (SANs, Lin et al., 2017) have advanced the state of the art on a variety of natural language processing (NLP) tasks, such as machine translation (Vaswani et al., 2017), seman- tic role labelling (Tan et al., 2018), and language representations (Devlin et al., 2018). However, a previous study empirically reveals that the hier- archical structure of the input sentence, which is essential for language understanding, is not well modeled by SANs (Tran et al., 2018). Recently, hybrid models which combine the strengths of SANs and recurrent neural networks (RNNs) have outperformed both individual architectures on a machine translation task (Chen et al., 2018). We attribute the improvement to that RNNs comple- ment SANs on the representation limitation of hi- ∗Work done when interning at Tencent AI Lab. erarchical structure, which is exactly the strength of RNNs (Tran et al., 2018). Starting with this intuition, we propose to fur- ther enhance the representational power of hybrid models with an advanced RNNs variant – Ordered Neurons LSTM (ON-LSTM, Shen et al., 2019). ON-LSTM is better at modeling hierarchical struc- ture by introducing a syntax-oriented inductive bias, which enables RNNs to perform tree-like composition by controlling the update frequency of neurons. Specifically, we stack SANs encoder on top of ON-LSTM encoder (cascaded encoder). SANs encoder is able to extract richer representa- tions from the input augmented with structure con- text. To reinforce the strength of modeling hierar- chical structure, we propose to simultaneously ex- pose both types of signals by explicitly combining outputs of the SANs and ON-LSTM encoders. We validate our hypothesis across a range of tasks, including machine translation, targeted lin- guistic evaluation, and logical inference. While machine translation is a benchmark task for deep learning …

Content truncated to 500 words