Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1099 - 1114 July 10-15, 2022 ©2022 Association for Computational Linguistics Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? Xiang Zhou Shiyue Zhang Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill {xzh, shiyue, mbansal}@cs.unc.edu Abstract Previous Part-Of-Speech (POS) induction models usually assume certain independence assumptions (e.g., Markov, unidirectional, lo- cal dependency) that do not hold in real lan- guages. For example, the subject-verb agree- ment can be both long-term and bidirectional. To facilitate flexible dependency modeling, we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent success of Masked Language Models (MLM). MPoSM can model arbitrary tag dependency and per- form POS induction through the objective of masked POS reconstruction. We achieve com- petitive results on both the English Penn WSJ dataset as well as the universal treebank con- taining 10 diverse languages. Though model- ing the long-term dependency should ideally help this task, our ablation study shows mixed trends in different languages. To better un- derstand this phenomenon, we design a novel synthetic experiment that can specifically diag- nose the model’s ability to learn tag agreement. Surprisingly, we find that even strong base- lines fail to solve this problem consistently in a very simplified setting: the agreement be- tween adjacent words. Nonetheless, MPoSM achieves overall better performance. Lastly, we conduct a detailed error analysis to shed light on other remaining challenges.1 1 Introduction Unsupervised Part-Of-Speech (POS) tagging is the task of discovering POS tags from text without any supervision. These unsupervised syntax induc- tion approaches can reduce the effort needed for collecting expensive syntactic annotation, and can bring us insights about what inductive bias leads to the emergence of syntax. Recent POS induction models have made great progress using different frameworks (Christodoulopoulos et al., 2010; Berg- Kirkpatrick et al., 2010; He et al., 2018; Stratos 1Our code is available at https://github.com/ owenzx/MPoSM DT NN IN DT NN VBZ RB JJ . Some areas of the factory are particularly dusty . DT NNS IN DT NN VBP RB JJ . Some news about the factory is particularly cheerful . Figure 1: Two long-term tag dependency examples in English. et al., 2016; Shi et al., 2020). However, most of them assume certain independence assumptions among POS tags, e.g., Markov (Merialdo, 1994; Berg-Kirkpatrick et al., 2010; Ammar et al., 2014; He et al., 2018), unidirectional (Tran et al., 2016), local dependency (Stratos et al., 2016; Gupta et al., 2022), etc. On the contrary, complex and long-term dependency appear in many real languages and plays an important role in defining the POS tags. For example, in Figure 1, the VBP tag of are and the NNS tag of areas depend on each other, and so do the VBZ tag of is and the NN tag of news.2 So in this case, models only conditioning the immediate preceding tag (Markov) or 1-2 neighboring words (local) cannot …

Content truncated to 500 words