Retrofitting Structure-aware Transformer Language Model for End Tasks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2151–2161, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2151 Retrofitting Structure-aware Transformer Language Model for End Tasks Hao Fei1, Yafeng Ren2∗and Donghong Ji1 1. Department of Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China 2. Guangdong University of Foreign Studies, China {hao.fei,renyafeng,dhji}@whu.edu.cn Abstract We consider retrofitting structure-aware Trans- former language model for facilitating end tasks by proposing to exploit syntactic dis- tance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task train- ing under multi-task learning scheme. Ex- perimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile in- ducing accurate syntactic phrases. By perform- ing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks. 1 Introduction Natural language models (LM) can generate fluent text and encode factual knowledge (Mikolov et al., 2013; Pennington et al., 2014; Merity et al., 2017). Recently, pre-trained contextualized language mod- els have given remarkable improvements on vari- ous NLP tasks (Peters et al., 2018; Radford et al., 2018; Howard and Ruder, 2018; Yang et al., 2019; Devlin et al., 2019; Dai et al., 2019). Among such methods, the Transformer-based (Vaswani et al., 2017) BERT has become a most popular encoder for obtaining state-of-the-art NLP task performance. It has been shown (Conneau et al., 2018; Tenney et al., 2019) that besides rich seman- tic information, implicit language structure knowl- edge can be captured by a deep BERT (Vig and Belinkov, 2019; Jawahar et al., 2019; Goldberg, 2019). However, such structure features learnt via the vanilla Transformer LM are insufficient for those NLP tasks that heavily rely on syntactic or linguistic knowledge (Hao et al., 2019). Some effort devote to improved the ability of structure ∗Corresponding author. (a) Full-layer (b) Middle-layer ystruc ytask ytask ystruc Figure 1: Full-layer multi-task learning for structural training (left), and the middle-layer training for deep structure-aware Transformer LM (right). learning in Transformer LM by installing novel syntax-attention mechanisms (Ahmed et al., 2019; Wang et al., 2019). Nevertheless, several limita- tions can be observed. First, according to the recent findings by probing tasks (Conneau et al., 2018; Tenney et al., 2019; Goldberg, 2019), the syntactic structure represen- tations are best retained right at the middle layers (Vig and Belinkov, 2019; Jawahar et al., 2019). Nevertheless, existing tree Transformers employ traditional full-scale training over the whole deep Transformer architecture (as shown in Figure 1(a)), consequently weakening the upper-layer semantic learning that can be crucial for end tasks. Sec- ond, these tree Transformer methods encode either standalone constituency or dependency structure, while different tasks can depend on varying types of structural knowledge. The constituent and depen- dency representation for syntactic structure share underlying linguistic characteristics, while the for- mer focuses on disclosing phrasal continuity and the latter aims at indicating …

Content truncated to 500 words