PaLM: A Hybrid Parser and Language Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3644–3651, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3644 PaLM: A Hybrid Parser and Language Model Hao Peng♠ Roy Schwartz♠♦ Noah A. Smith♠♦ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen Institute for Artificial Intelligence {hapeng,roysch,nasmith}@cs.washington.edu Abstract We present PaLM, a hybrid parser and neu- ral language model. Building on an RNN lan- guage model, PaLM adds an attention layer over text spans in the left context. An unsuper- vised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it out- performs strong baselines. If syntactic anno- tations are available, the attention component can be trained in a supervised manner, pro- viding syntactically-informed representations of the context, and further improving language modeling performance. 1 Introduction Recent language models have shown very strong data-fitting performance (Jozefowicz et al., 2016; Merity et al., 2018). They offer useful products in- cluding, most notably, contextual embeddings (Pe- ters et al., 2018; Radford et al., 2019), which benefit many NLP tasks such as text classifica- tion (Howard and Ruder, 2018) and dataset cre- ation (Zellers et al., 2018). Language models are typically trained on large amounts of raw text, and therefore do not ex- plicitly encode any notion of structural informa- tion. Structures in the form of syntactic trees have been shown to benefit both classical NLP models (Gildea and Palmer, 2002; Punyakanok et al., 2008; Das et al., 2012, inter alia) and recent state-of-the-art neural models (Dyer et al., 2016; Swayamdipta et al., 2018; Peng et al., 2018b; Strubell et al., 2018, inter alia). In this paper we show that LMs can benefit from syntactically- inspired encoding of the context. We introduce PaLM (parser and language model; Fig. 1), a novel hybrid model combin- ing an RNN language model with a constituency parser. The LM in PaLM attends over spans of EMNLP IJCNLP 2019 will be held in xt+1? [EMNLP IJCNLP 2019] [will be held] in [[EMNLP IJCNLP] [2019]] [[will be] [held]] in [[[EMNLP][IJCNLP]][2019]][[[will] [be]] [held]] in LM attends over spans Parser recursively splits spans by attention weights Figure 1: An illustration of PaLM. The LM (first line) predicts the next word (xt+1, double blue arrow) by attending over previous spans ending in time t − 1 (dashed lines). The parser (lines 2–4) splits the pre- fix into two spans (line 2) by the taking the top scoring attended span (red solid line) and the prefix leading to it. It then recursively splits the two sub-spans using the same procedure (line 3). Finally, spans of length two are trivially split into terminal nodes (line 4). tokens, implicitly learning which syntactic con- stituents are likely. A span-based parser is then de- rived from the attention information (Stern et al., 2017). PaLM has several benefits. First, it is an …

Content truncated to 500 words