Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

Findings of the Association for Computational Linguistics: ACL 2022, pages 3276 - 3290 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging Houquan Zhou, Yang Li∗, Zhenghua Li�, Min Zhang Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China {hqzhou,ylinlp}@stu.suda.edu.cn; {zhli13,minzhang}@suda.edu.cn Abstract In recent years, large-scale pre-trained lan- guage models (PLMs) have made extraordi- nary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utiliz- ing PLMs are few and fail to achieve state- of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, mak- ing it very challenging to incorporate contex- ualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tag- ging. The discriminative encoder of CRF-AE can straightforwardly incorporate PLM word representations. Moreover, inspired by feature- rich HMM, we reintroduce hand-crafted fea- tures into the decoder of CRF-AE. Finally, ex- periments clearly show that our model outper- forms previous state-of-the-art models by a large margin on Penn Treebank and multilin- gual Universal Dependencies treebank v2.0. 1 Introduction Unsupervised learning has been an important yet challenging research direction in NLP (Klein and Manning, 2004; Liang et al., 2006; Seginer, 2007). Training models directly from unlabeled data can relieve painful data annotation and is thus espe- cially attractive for low-resource languages (He et al., 2018). As three typical tasks related to syn- tactic analysis, unsupervised part-of-speech (POS) tagging (or induction), dependency parsing, and constituency parsing have attracted intensive in- terest during the past three decades (Pereira and Schabes, 1992; Christodoulopoulos et al., 2010, in- ter alia). Compared with tree-structure dependency and constituency parsing, POS tagging corresponds ∗ Houquan and Yang make equal contributions to this work. Zhenghua is the corresponding author. I looked at my watch . PRP VBD IN PRP$ NN . Figure 1: Example of POS tagging. to simpler sequential structure, and aims to assign a POS tag to each word, as depicted in Figure 1. Be- sides the alleviation of labeled data, unsupervised POS tagging is particularly valuable for child lan- guage acquisition study because every child man- ages to induce syntactic categories without access to labeled data (Yuret et al., 2014). Nowadays, supervised POS tagging models trained on large-scale labeled data can already achieve extremely high accuracy, for example over 97.5% on English Penn Treebank (PTB) texts (Huang et al., 2015; Bohnet et al., 2018; Zhou et al., 2020). However, unsupervised POS tagging, though having attracted a lot of research interest (Lin et al., 2015; Tran et al., 2016; He et al., 2018; Stratos, 2019; Gupta et al., 2020), can only achieve at most 80.8% many-to-one (M-1) accuracy, where M-1 means multiple induced tags can be mapped to a single ground-truth tag when evaluating the model on the test data. The …

Content truncated to 500 words