Neural Bi-Lexicalized PCFG Induction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2688–2699 August 1–6, 2021. ©2021 Association for Computational Linguistics 2688 Neural Bi-Lexicalized PCFG Induction Songlin Yang♣, Yanpeng Zhao♦, Kewei Tu♣˚ ♣School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences ♦ILCC, University of Edinburgh {yangsl,tukw}@shanghaitech.edu.cn yannzhao.ed@gmail.com Abstract Neural lexicalized PCFGs (L-PCFGs) (Zhu et al., 2020) have been shown effective in grammar induction. However, to reduce com- putational complexity, they make a strong in- dependence assumption on the generation of the child word and thus bilexical dependen- cies are ignored. In this paper, we propose an approach to parameterize L-PCFGs with- out making implausible independence assump- tions. Our approach directly models bilexi- cal dependencies and meanwhile reduces both learning and representation complexities of L- PCFGs. Experimental results on the English WSJ dataset confirm the effectiveness of our approach in improving both running speed and unsupervised parsing performance. 1 Introduction Probabilistic context-free grammars (PCFGs) has been an important probabilistic approach to syntac- tic analysis (Lari and Young, 1990; Jelinek et al., 1992). They assign a probability to each of the parses admitted by CFGs and rank them by the plau- sibility in such a way that the ambiguity of CFGs can be ameliorated. Still, due to the strong indepen- dence assumption of CFGs, vanilla PCFGs (Char- niak, 1996) are far from adequate for highly am- biguous text. A common premise for tackling the issue is to incorporate lexical information and weaken the in- dependence assumption. There have been many ap- proaches proposed under the premise (Magerman, 1995; Collins, 1997; Johnson, 1998; Klein and Manning, 2003). Among them lexicalized PCFGs (L-PCFGs) are a relatively straightforward formal- ism (Collins, 2003). L-PCFGs extend PCFGs by associating a word, i.e., the lexical head, with each grammar symbol. They can thus exploit lexical ˚Corresponding Author information to disambiguate parsing decisions and are much more expressive than vanilla PCFGs. However, they suffer from representation and in- ference complexities. For representation, the ad- dition of lexical information greatly increases the number of parameters to be estimated and exac- erbates the data sparsity problem during learning, so the expectation-maximisation (EM) based esti- mation of L-PCFGs has to rely on sophisticated smoothing techniques and factorizations (Collins, 2003). As for inference, the CYK algorithm for L-PCFGs has a Opl5|G|q complexity, where l is the sentence length and |G| is the grammar con- stant. Although Eisner and Satta (1999) manage to reduce the complexity to Opl4|G|q, inference with L-PCFGs is still relatively slow, making them less popular nowadays. Recently, Zhu et al. (2020) combine the ideas of factorizing the binary rule probabilities (Collins, 2003) and neural parameterization (Kim et al., 2019) and propose neural L-PCFGs (NL-PCFGs), achieving good results in both unsupervised depen- dency and constituency parsing. Neural parame- terization is the key to success, which …

Content truncated to 500 words