Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 409–424 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 409 Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads Bowen Li† Taeuk Kim‡ Reinald Kim Amplayo† Frank Keller† †ILCC, School of Informatics, University of Edinburgh, UK ‡Dept. of Computer Science and Engineering, Seoul National University, Korea {bowen.li,reinald.kim}@ed.ac.uk taeuk@europa.snu.ac.kr keller@inf.ed.ac.uk Abstract Transformer-based pre-trained language mod- els (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous ap- proaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing ap- proach that extracts constituency trees from PLM attention heads. We rank transformer at- tention heads based on their inherent proper- ties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outper- form existing approaches if there is no devel- opment set present. Our unsupervised parser can also be used as a tool to analyze the gram- mars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank. 1 Introduction Pre-trained language models (PLMs), particularly BERT (Devlin et al., 2019) and others (Yang et al., 2019; Liu et al., 2019b; Radford et al., 2019) based on the transformer architecture (Vaswani et al., 2017), have dramatically improved the state of the art in NLP. Such models make it possible to train a large, generic language model on vast unannotated datasets, and then fine-tune it for a specific task us- ing a small amount of annotated data. The success of PLMs has led to a large literature investigating the linguistic knowledge that PLMs learn implicitly during pre-training (Liu et al., 2019a; Clark et al., 2019; Kovaleva et al., 2019; Pimentel et al., 2020), sometimes referred to as BERTology (Rogers et al., 2020). BERTology has been particularly concerned with the question whether BERT-type models learn syn- tactic structure. Typical approaches include test suites of sentences that instantiate specific syntactic structures (Goldberg, 2019), general probes (also known as diagnostic classifiers, Belinkov and Glass 2019) or structural probes (Hewitt and Manning, 2019). All of these approaches are limited: the first one requires the laborious compilation of language- and construction-specific suites of sentences; the second one sometimes fails to adequately reflect differences in representations (Zhang and Bowman, 2018; Hewitt and Liang, 2019; Voita and Titov, 2020); the third one involves designing a novel ex- traction model that is not applicable to tasks other than probing (Maudslay et al., 2020). It is therefore natural to use a parsing task to test whether the representations …

Content truncated to 500 words