Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6611–6628 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6611 Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach Wenyu Du1,2∗, Zhouhan Lin3,4∗, Yikang Shen3,4, Timothy J. O’Donnell3,5,6, Yoshua Bengio3,4 and Yue Zhang1,2 1School of Engineering, Westlake University 2Institute of Advanced Technology, Westlake Institute for Advanced Study 3Mila 4Universit´e de Montr´eal 5Department of Linguistics, McGill University 6Canada CIFAR AI Chair Abstract It is commonly believed that knowledge of syn- tactic structure should improve language mod- eling. However, effectively and computation- ally efficiently incorporating syntactic struc- ture into neural language models has been a challenging topic. In this paper, we make use of a multi-task objective, i.e., the models si- multaneously predict words as well as ground truth parse trees in a form called “syntactic distances”, where information between these two separate objectives shares the same inter- mediate representation. Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training sig- nals, the model is able to achieve lower per- plexity and induce trees with better quality. 1 Introduction It is widely believed in linguistics, cognitive sci- ence, and computational linguistics that the la- tent structure underlying how words combine to form sentences is best represented as a tree struc- ture. The study of the computational mechanisms and systems of constraints that characterize such derivations or parse trees is a central question in these fields (Pollard and Sag, 1994; Steedman and Baldridge, 2011; Huddleston and Pullum, 2002; Adger, 2003; Bresnan, 2001; Chomsky, 1995; Sag et al., 2003). Using syntactic information for the language modeling task has been a popular research topic since the 1990s. Early efforts included various approaches that attempted to incorporate shallow syntactic information such as POS tags (Heeman and Allen, 1997; Srinivas, 1996) as well as a more complete structures (Wright et al., 1994; Jurafsky et al., 1995). Most of such work falls under the topic of structured language modeling (Chelba and ∗Equal contribution. Jelinek, 2000; Van Uytsel et al., 2001; Xu et al., 2002). With the resurgence of neural network ap- proaches, sequential, large-scale neural language models have been shown to significantly outper- form traditional language models (Merity et al., 2017; Yang et al., 2018) without using syntactic structural information. On another scenario, recent analysis also reveals that state-of-the-art sequential neural language models still fail to learn certain long-range syntactic dependencies (Kuncoro et al., 2018). Thus it is an interesting problem to explore the relation between language models and syntax and investigate whether syntax can be integrated to enhance neural language models. To this end, two main lines of work have been investigated, namely transition-based and distance- based methods, respectively. The former strand of work has sought to jointly train a transition-based parser (Nivre, 2008; Zhang and Nivre, 2011; An- dor et al., 2016) with a language model using a linearized structured sentence. For example, recur- rent neural network …

Content truncated to 500 words