Two Discourse Driven Language Models for Semantics

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 290–300, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Two Discourse Driven Language Models for Semantics Haoruo Peng and Dan Roth University of Illinois, Urbana-Champaign Urbana, IL, 61801 {hpeng7,danr}@illinois.edu Abstract Natural language understanding often re- quires deep semantic knowledge. Ex- panding on previous proposals, we suggest that some important aspects of semantic knowledge can be modeled as a language model if done at an appropriate level of ab- straction. We develop two distinct mod- els that capture semantic frame chains and discourse information while abstract- ing over the specific mentions of predi- cates and entities. For each model, we in- vestigate four implementations: a “stan- dard” N-gram language model and three discriminatively trained “neural” language models that generate embeddings for se- mantic frames. The quality of the se- mantic language models (SemLM) is eval- uated both intrinsically, using perplexity and a narrative cloze test and extrinsically – we show that our SemLM helps improve performance on semantic natural language processing tasks such as co-reference res- olution and discourse parsing. 1 Introduction Natural language understanding often necessitates deep semantic knowledge. This knowledge needs to be captured at multiple levels, from words to phrases, to sentences, to larger units of dis- course. At each level, capturing meaning fre- quently requires context sensitive abstraction and disambiguation, as shown in the following exam- ple (Winograd, 1972): Ex.1 [Kevin] was robbed by [Robert]. [He] was arrested by the police. Ex.2 [Kevin] was robbed by [Robert]. [He] was rescued by the police. In both cases, one needs to resolve the pronoun “he” to either “Robert” or “Kevin”. To make the correct decisions, one needs to know that the subject of “rob” is more likely than the object of “rob” to be the object of “arrest” while the object of “rob” is more likely to be the object of “rescue”. Thus, beyond understanding individual predicates (e.g., at the semantic role labeling level), there is a need to place them and their arguments in a global context. However, just modeling semantic frames is not sufficient; consider a variation of Ex.1: Ex.3 Kevin was robbed by Robert, but the police mistakenly arrested him. In this case, “him” should refer to “Kevin” as the discourse marker “but” reverses the meaning, illustrating that it is necessary to take discourse markers into account when modeling semantics. In this paper we propose that these aspects of semantic knowledge can be modeled as a Seman- tic Language Model (SemLM). Just like the “stan- dard” syntactic language models (LM), we de- fine a basic vocabulary, a finite representation lan- guage, and a prediction task, which allows us to model the distribution over the occurrence of el- ements in the vocabulary as a function of their (well-defined) context. In difference from syn- tactic LMs, we represent natural language at a higher level of semantic abstraction, thus facilitat- ing modeling deep semantic knowledge. We propose two distinct discourse driven lan- guage models …

Content truncated to 500 words