Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations

Proceedings of the 28th International Conference on Computational Linguistics, pages 3806–3817 Barcelona, Spain (Online), December 8-13, 2020 3806 Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations Ying Li1, Zhenghua Li1∗and Min Zhang1 1Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China yingli hlt@foxmail.com, {zhli13, minzhang}@suda.edu.cn Abstract In recent years, parsing performance is dramatically improved on in-domain texts thanks to the rapid progress of deep neural network models. The major challenge for current parsing research is to improve parsing performance on out-of-domain texts that are very different from the in- domain training data when there is only a small-scale out-domain labeled data. To deal with this problem, we propose to improve the contextualized word representations via adversarial learning and fine-tuning BERT processes. Concretely, we apply adversarial learning to three rep- resentative semi-supervised domain adaption methods, i.e., direct concatenation (CON), feature augmentation (FA), and domain embedding (DE) with two useful strategies, i.e., fused target- domain word representations and orthogonality constraints, thus enabling to model more pure yet effective domain-specific and domain-invariant representations. Simultaneously, we utilize a large-scale target-domain unlabeled data to fine-tune BERT with only the language model loss, thus obtaining reliable contextualized word representations that benefit for the cross-domain de- pendency parsing. Experiments on a benchmark dataset show that our proposed adversarial ap- proaches achieve consistent improvements, and fine-tuning BERT further boosts the parsing ac- curacy by a large margin. Our single model achieves the same state-of-the-art performance as the top submitted system in the NLPCC-2019 shared task, which uses ensemble models and BERT. 1 Introduction Dependency parsing aims to capture syntax with a dependency tree and is proven to be helpful for various natural language processing (NLP) tasks, such as semantic role labeling (Xia et al., 2019), natural language generation (Park and Kang, 2019), and machine translation (Hadiwinoto and Ng, 2017). Given an input sentence s = w1w2 . . . wn, a dependency tree, as depicted in Figure 1, is defined as d = {(h, m, l), 0 ≤ h ≤ n, 1 ≤ m ≤ n, l ∈ L}, where (h, m, l) is a dependency from the head word wh to the child word wm with the relation label l ∈ L, and w0 is a pseudo word that points to the root word of the sentence. In recent years, neural network based approaches have achieved remarkable improvement and outper- formed the traditional discrete-feature based approaches by a large margin in dependency parsing (Chen and Manning, 2014; Kiperwasser and Goldberg, 2016; Andor et al., 2016; Dozat and Manning, 2017). Most remarkably, Dozat and Manning (2017) propose a simple yet effective deep BiAffine parser and achieve the state-of-the-art accuracy on a variety of datasets and languages. However, the domain adaptation problem, i.e., how to improve parsing performance on texts that are very different from the training data, remains a key challenge for the parsing community, especially when trying to apply the parsing technique to real-life web data. Taking the examples in …

Content truncated to 500 words