Unsupervised Dependency Graph Network

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4767 - 4784 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Unsupervised Dependency Graph Network Yikang Shen1,2, Shawn Tan1, Alessandro Sordoni3, Peng Li4, Jie Zhou2, Aaron Courville1 1Mila/Université de Montréal 2Pattern Recognition Center, WeChat AI, Tencent Inc 3Microsoft Research Montréal 4Institute for AI Industry Research (AIR), Tsinghua University yikang.shn@gmail.com Abstract Recent work has identified properties of pre- trained self-attention models that mirror those of dependency parse structures. In partic- ular, some self-attention heads correspond well to individual dependency types. In- spired by these developments, we propose a new competitive mechanism that encourages these attention heads to model different depen- dency relations. We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency struc- tures from raw corpora and the masked lan- guage modeling task. Experiment results show that UDGN achieves very strong unsupervised dependency parsing performance without gold POS tags and any other external information. The competitive gated heads show a strong correlation with human-annotated dependency types. Furthermore, the UDGN can also achieve competitive performance on masked language modeling and sentence textual sim- ilarity tasks 1. 1 Introduction The goal of unsupervised dependency parsing is to induce dependency grammar from corpora that don’t have annotated parse trees (Marecek, 2016). Although the task is difficult, one advantage of unsupervised methods is that they can leverage vast amount of unannotated raw corpus (Han et al., 2020). Thus, adapting the task into a pretrain- ing framework is increasingly tempting. The in- duced dependency trees can also help solve other NLP problems such as unsupervised discourse pars- ing (Nishida and Nakayama, 2020), aspect-based sentiment analysis (Dai et al., 2021) and intent dis- covery (Liu et al., 2021). Furthermore, the task can also be used as a probe to verify cognitive theories for human language acquistition (Yang et al., 2020; 1Our code is publicly available at https://github. com/yikangshen/UDGN. Figure 1: The architecture of Unsupervised Depen- dency Graph Network (UDGN). Given an input sen- tence, the parser can predict the dependency relation between tokens and generate a soft mask to approxi- mate the undirected dependency graph. Conditioning on the mask, the DGN computes contextual word em- beddings for the training task. Since the mask is soft, the gradient can be backpropagated from the DGN into the parser. Thus UDGN can induce grammar while training on masked language modeling or other down- stream tasks. Pate and Goldwater, 2013; Katzir, 2014; Solan et al., 2002). Pretrained Language Models (PLMs) have be- come the foundation of modern natural language processing in the last few years (Bommasani et al., 2021). They dominate the most if not all NLP tasks. But Recent works show that, beyond pre- training big models on large-scale corpora, deep learning methods can improve performance by in- creasing models’ awareness of syntactic informa- tion (Kuncoro et al., 2020). These methods either include known structural information as input to the model (Sundararaman et al., 2019; Bai et …

Content truncated to 500 words