Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 183–193 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 183 Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP Hao Fei1, Yafeng Ren2 and Donghong Ji1∗ 1. Department of Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China 2. Guangdong University of Foreign Studies, China {hao.fei,renyafeng,dhji}@whu.edu.cn Abstract Syntax has been shown useful for various NLP tasks, while existing work mostly encodes sin- gleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledge Dis- tillation, to integrate heterogeneous structure knowledge into a unified sequential LSTM en- coder. Experimental results on four typical syntax-dependent tasks show that our method outperforms tree encoders by effectively in- tegrating rich heterogeneous structure syntax, meanwhile reducing error propagation, and also outperforms ensemble methods, in terms of both the efficiency and accuracy. 1 Introduction Integrating syntactic information into neural net- works has received increasing attention in natu- ral language processing (NLP), which has been used for a wide range of end tasks, such as senti- ment analysis (SA) (Nguyen and Shirai, 2015; Teng and Zhang, 2017; Looks et al., 2017; Zhang and Zhang, 2019), neural machine translation (NMT) (Cho et al., 2014; Garmash and Monz, 2015; G¯u et al., 2018), language modeling (Yazdani and Hen- derson, 2015; Zhang et al., 2016; Zhou et al., 2017), semantic role labeling (SRL) (Marcheggiani and Titov, 2017; Strubell et al., 2018; Fei et al., 2020c), natural language inference (NLI) (Tai et al., 2015a; Liu et al., 2018) and text classification (Chen et al., 2015; Zhang et al., 2018b). Despite the usefulness of structure knowledge, most existing models use only a single syntactic tree, such as a constituency or a dependency tree. Constituent and dependency representation for syntactic structure share underlying linguistic and computational characteristics, while differ also in various aspects. For example, the former focuses ∗Corresponding author. amod nmod nmod nmod case dobj nsubj John visited his brother at school last week NNP VBD PRP NN IN NN JJ NN NP VP NP PP NP S A0 visit.01 A1 AM-LOC AM-TMP (2) (3) (4) (1) Figure 1: An example illustrating the mutual benefit of constituency and dependency tree structures. (1) refers to the constituency tree structure, (2) indicates the se- mantic role labels, (3) refers to the example sentence, (4) represents the dependency tree structure. on revealing the continuity of phrases, while the latter is more effective in representing the depen- dencies among elements. By integrating the two representations from heterogeneous trees, the mu- tual benefit has been explored for joint parsing tasks (Collins, 1997; Charniak and Johnson, 2005; Farkas et al., 2011; Yoshikawa et al., 2017; Zhou and Zhao, 2019). Intuitively, complementary ad- vantages from heterogeneous trees can facilitate a range of NLP tasks, especially syntax-dependent ones such as SRL and NLI. Taking the sentence of Figure 1 as example, where an example is shown from …

Content truncated to 500 words