Co-training an Unsupervised Constituency Parser with Weak Supervision

Findings of the Association for Computational Linguistics: ACL 2022, pages 1274 - 1291 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Co-training an Unsupervised Constituency Parser with Weak Supervision Nickil Maveli and Shay B. Cohen Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB n.maveli@sms.ed.ac.uk, scohen@inf.ed.ac.uk Abstract We introduce a method for unsupervised pars- ing that relies on bootstrapping classifiers to identify if a node dominates a specific span in a sentence. There are two types of classifiers, an inside classifier that acts on a span, and an outside classifier that acts on everything out- side of a given span. Through self-training and co-training with the two classifiers, we show that the interplay between them helps improve the accuracy of both, and as a result, effec- tively parse. A seed bootstrapping technique prepares the data to train these classifiers. Our analyses further validate that such an ap- proach in conjunction with weak supervision using prior branching knowledge of a known language (left/right-branching) and minimal heuristics injects strong inductive bias into the parser, achieving 63.1 F1 on the English (PTB) test set. In addition, we show the effectiveness of our architecture by evaluating on treebanks for Chinese (CTB) and Japanese (KTB) and achieve new state-of-the-art results.1 1 Introduction Pre-trained language models (PLMs) have become a standard tool in the Natural Language Process- ing (NLP) toolkit, offering the benefits of learning from large amounts of unlabeled data while pro- viding modular function in many NLP tasks that require supervision. Recent work has shown that PLMs capture different types of linguistic regulari- ties and information, for instance, the lower layers capture phrase-level information which becomes less prominent in the upper layers (Jawahar et al., 2019), span representations constructed from these models can encode rich syntactic phenomena, like the ability to track subject-verb agreement (Gold- berg, 2019), dependency trees can be embedded 1Our code and pre-trained models are available at https://github.com/Nickil21/ weakly-supervised-parsing. Figure 1: A depiction of a syntax tree, with the inside string as depicted by the sequence xi · · · xj and the outside string as depicted by the sequence (x1 · · · xi−1, xj+1 · · · xn) that provides external con- text for the inside representations. within the geometry of BERT’s hidden states (He- witt and Manning, 2019), and most relevantly to this paper, syntactic information via self-attention mechanisms (Wang et al., 2019; Kim et al., 2020). We offer another perspective on the way PLMs represent syntactic information. We demonstrate the usability of PLMs to capture syntactic informa- tion by developing an unsupervised parsing model that makes heavy use of PLMs. The learning algo- rithm is light in the injection of hard bias to parse text, emphasizing the role of PLMs in capturing syntactic information. Our approach to unsupervised parsing is inspired by recent work in the area of spectral learning for parsing (Cohen et al., 2014, 2013) and unsuper- vised estimation of probabilistic context-free gram- mars (PCFGs; Clark …

Content truncated to 500 words