Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1660–1672 June 6–11, 2021. ©2021 Association for Computational Linguistics 1660 Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle Yikang Shen Mila/Université de Montréal yi-kang.shen@umontreal.ca Shawn Tan Mila/Université de Montréal tanjings@mila.quebec Alessandro Sordoni Microsoft Research Montréal Siva Reddy Mila/McGill University Aaron Courville Mila/Université de Montréal Abstract Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the struc- ture with an incremental parser and maintains the conditional probability setting of a stan- dard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing and syntactic generaliza- tion tests, while using fewer parameters than other models. 1 Introduction Several recent works have systematically studied the linguistic abilities of modern language models, particularly syntax (Linzen et al., 2016; Marvin and Linzen, 2018; Gulordava et al., 2018). They find that most language models are good at capturing frequent syntactic structures but do not generalize well to those in the long tail. Moreover, although some excel at having low perplexity scores, this is less due to their syntactic ability but more due to capturing collocations (frequently co-occurring words). Recently, Hu et al. (2020) show that RNNs underperform on a syntactic generalization (SG) test set, whereas models that have an explicit notion of syntax, such as RNNG (Dyer et al., 2016), fare well on SG but at the cost of generally poorer lan- guage modeling (higher perplexity). Transformer- based models achieve strong performance when trained with large datasets, but are worse than ran- dom when trained on a small dataset. These works showed that building language models with an explicit internal model of syntax Figure 1: The mechanism of SOM. “Context” is a dis- tributed representation of previous sentences. It could also represents the source sentence in a sequence to se- quence task. It incrementally build subtrees given the input sentences. A RNN will takes the context represen- tation and the representations of subtrees in the current sentence to predict next token. helps in achieving better performance in SG tasks and is also thought to help learn more efficiently in low data settings. However, building syntax- aware models that also obtain strong language mod- eling performance, when compared with recent transformer-based models, has until now seemed elusive. In this work, we propose a new syntax- aware language model dubbed Syntactic Ordered Memory (SOM; Fig. 1), which jointly acts as a language model and an incremental parser. SOM inherits the syntax representation used in Ordered Memory (OM; Shen et al. 2019) in which syntax trees are embedded in …

Content truncated to 500 words