Inter-document Contextual Language model

Proceedings of NAACL-HLT 2016, pages 762–766, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Inter-document Contextual Language Model Quan Hung Tran and Ingrid Zukerman and Gholamreza Haffari Faculty of Information Technology Monash University, Australia hung.tran,ingrid.zukerman,gholamreza.haffari@monash.edu Abstract In this paper, we examine the impact of employing contextual, structural information from a tree-structured document set to derive a language model. Our results show that this in- formation significantly improves the accuracy of the resultant model. 1 Introduction Conventional Language Models (LMs) are based on n-grams, and thus rely upon a limited number of preceding words to assign a probability to the next word in a document. Recently, Mikolov et al. (2010) proposed a Recurrent Neural Network (RNN) LM which uses a vector representation of all the pre- ceding words in a sentence as the context for lan- guage modeling. This model, which theoretically can utilize an infinite context window within a sen- tence, yields an LM with lower perplexity than that of n-gram-based LMs. However, the model does not leverage the wider contextual information provided by words in other sentences in a document or in re- lated documents. Several researchers have explored extending the contextual information of an RNN-based LM. Mikolov and Zweig (2012) proposed a context- dependent RNN LM that employs Latent Dirich- let Allocation for modeling a long span of context. Wang and Cho (2015) offered a bag-of-words repre- sentation of preceding sentences as the context for the RNN LM. Ji et al. (2015) used a Document- Context LM (DCLM) to leverage both intra- and inter-sentence context. These works focused on contextual information at the document level for LM, but did not con- sider information at the inter-document level. Many document sets on the Internet are structured, which means there are connections between different docu- ments. This phenomenon is prominent in social me- dia, where all the posts are directly linked to several other posts. We posit that these related documents could hold important information about a particu- lar post, including the topic and language use, and propose an RNN-based LM architecture that utilizes both intra- and inter-document contextual informa- tion. Our approach, which was tested on the social media dataset reddit, yielded promising results, which significantly improve on the state of the art. 2 Dataset We used pre-collected reddit data,1 which as of December, 2015, consists of approximately 1.7 bil- lion comments in JSON format. A comment thread starts with a “topic”, which might be a link or an im- age. The users then begin to comment on the topic, or reply to previous comments. Over time, this pro- cess creates a tree-structured document repository (Figure 1), where a level indicator is assigned to each comment, e.g., a response to the root topic is assigned level 1, and the reply to a level n com- ment is assigned level n + 1. We parsed the raw data in JSON format into a tree structure, removing threads that have less than three comments, contain deleted comments, or …

Content truncated to 500 words