Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing

Proceedings of the 29th International Conference on Computational Linguistics, pages 5438–5449 October 12–17, 2022. 5438 Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing Yuanhe Tian♥, Yan Song♠†, Fei Xia♥ ♥University of Washington ♠University of Science and Technology of China ♥{yhtian, fxia}@uw.edu ♠clksong@gmail.com Abstract Dependency parsing is an important funda- mental natural language processing task which analyzes the syntactic structure of an input sen- tence by illustrating the syntactic relations be- tween words. To improve dependency pars- ing, leveraging existing dependency parsers and extra data (e.g., through semi-supervised learning) has been demonstrated to be effec- tive, even though the final parsers are trained on inaccurate (but massive) data. In this pa- per, we propose a frustratingly easy approach to improve graph-based dependency parsing, where a structure-aware encoder is pre-trained on auto-parsed data by predicting the word de- pendencies and then fine-tuned on gold depen- dency trees, which differs from the usual pre- training process that aims to predict the con- text words along dependency paths. Experi- mental results and analyses demonstrate the effectiveness and robustness of our approach to benefit from the data (even with noise) processed by different parsers, where our ap- proach outperforms strong baselines under dif- ferent settings with different dependency stan- dards and model architectures used in pre- training and fine-tuning. More importantly, further analyses find that only 2K auto-parsed sentences are required to obtain improvement when pre-training vanilla BERT-large based parser without requiring extra parameters.1 1 Introduction Dependency parsing aims to produce the syntactic structure of a sentence by illustrating the syntac- tic relations between words, where the words with dependency relations are connected by directed and labeled arcs. It is an important fundamen- tal natural language processing (NLP) task that is widely used to enhance downstream NLP tasks (Cai et al., 2009; Strubell et al., 2018; Huang and †Corresponding author. 1Our code is available at https://github.com/ synlp/DMPar. Carley, 2019; Zhang et al., 2019; Guo et al., 2019; Nie et al., 2020; Zhou et al., 2020b; Chen et al., 2020; Tian et al., 2022) such as coreference resolu- tion, relation extraction, and sentiment analysis. To produce the dependency structure of a sen- tence, the contextual information is of great impor- tance to achieve good model performance. Thus, most recent studies (Dozat and Manning, 2017; Zhou and Zhao, 2019; Zhou et al., 2020a,b; Mrini et al., 2020; Zhang et al., 2021) leverage advanced encoders (e.g., bi-LSTM, Transformer (Vaswani et al., 2017)) to model the contextual information of the input and obtain outstanding performance. In addition, because leveraging different models to obtain better results is an important technique for many NLP tasks (Juraska et al., 2018; Kobayashi, 2018; Kuwabara et al., 2020; Qin et al., 2021), many previous studies apply this technique to de- pendency parsing to further improve model per- formance. Under this paradigm, many studies uti- lize semi-supervised methods (e.g., self-training) to benefit from auto-processed extra data which is used to extract useful features (Smith and Eisner, 2007; Koo et al., 2008; Bansal …

Content truncated to 500 words