Multi-Layer Pseudo-Siamese Biaffine Model for Dependency Parsing

Proceedings of the 29th International Conference on Computational Linguistics, pages 5476–5487 October 12–17, 2022. 5476 Multi-Layer Pseudo-Siamese Biaffine Model for Dependency Parsing Ziyao Xu1, Houfeng Wang1, Bingdong Wang2 1MOE Key Lab of Computational Linguistics, Peking University 2Beijing Huilan Technology Co., Ltd. {xzyxzy,wanghf}@pku.edu.cn, wangbd@huilan.com Abstract Biaffine method is a strong and efficient method for graph-based dependency parsing. However, previous work only used the biaffine method at the end of the dependency parser as a scorer, and its application in multi-layer form is ig- nored. In this paper, we propose a multi-layer pseudo-Siamese biaffine model for neural de- pendency parsing. In this model, we modify the biaffine method so that it can be utilized in multi-layer form, and use pseudo-Siamese biaffine module to construct arc weight matrix for final prediction. In our proposed multi-layer architecture, the biaffine method plays impor- tant roles in both scorer and attention mecha- nism at the same time in each layer. We eval- uate our model on PTB, CTB, and UD. The model achieves state-of-the-art results on these datasets. Further experiments show the benefits of introducing multi-layer form and pseudo- Siamese module into the biaffine method with low efficiency loss. 1 Introduction Dependency parsing is a fundamental task in NLP. Given a input sequence s = w0w1...wn, the out- put is a dependency tree t = {(h, d, l), 0 ≤ h ≤ n, 1 ≤ d ≤ n, l ∈ L}, where wi(1 ≤ i ≤ n) is a word, w0 is a pseudo-word as root and (h, d, l) is an arc from wh to wd with label l in a relation set L. Due to the simplicity and effectiveness of representing syntactic information by using the tree structure, many works involve dependency parsing, such as syntax-enhanced pre-trained model (Xu et al., 2021). There are two approaches to dependency parsing: transition-based and graph-based methods. Graph- based dependency parsing scores the components of a sentence and selects the highest scoring tree. Dozat and Manning (2017) for the first time intro- duce biaffine method into the dependency parsing. This strong and efficient biaffine first-order graph- based parser consists of BiLSTM encoder and bi- affine scorer. After the biaffine parser is proposed, many works have focused on further improving the performance of this parser. Since the biaffine parser consists of two parts: BiLSTM encoder and biaffine scorer, there are two main directions of im- provement focusing on the two parts respectively. One direction of improvement is modifying the encoder. Straka (2018) introduce pre-trained model in the embedding stage. Na et al. (2019) fuse the hidden states of different layers of BiLSTM to construct the input of the biaffine scorer. Ji et al. (2019) add graph neural networks before the bi- affine scorer to capture high-order information. Li et al. (2019) apply the self-attention mechanism as the replacement of BiLSTM encoder. Mrini et al. (2020) introduce the Label Attention Layer, a new form of self-attention where attention heads represent labels, into the encoder. In these works, although the encoder …

Content truncated to 500 words