The Context-Dependent Additive Recurrent Neural Net

Proceedings of NAACL-HLT 2018, pages 1274–1283 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics The Context-dependent Additive Recurrent Neural Net Quan Hung Tran 1,2 Tuan Manh Lai 2 Gholamreza Haffari 1 Ingrid Zukerman 1 Trung Bui 2 Hung Bui 3 1 Monash University, Clayton, Australia 2 Adobe Research, San Jose, CA 3 DeepMind, Mountain View, CA Abstract Contextual sequence mapping is one of the fundamental problems in Natural Language Processing. Instead of relying solely on the information presented in a text, the learning agents have access to a strong external signal given to assist the learning process. In this pa- per, we propose a novel family of Recurrent Neural Network unit: the Context-dependent Additive Recurrent Neural Network (CARNN) that is designed specifically to leverage this ex- ternal signal. The experimental results on pub- lic datasets in the dialog problem (Babi dialog Task 6 and Frame), contextual language model (Switchboard and Penn Discourse Tree Bank) and question answering (TrecQA) show that our novel CARNN-based architectures outper- form previous methods. 1 Introduction Sequence mapping is one of the most prominent class of problems in Natural Language Processing (NLP). This is due to the fact that written language is sequential in nature. In English, a word is a se- quence of characters, a sentence is a sequence of words, a paragraph is a sequence of sentences, and so on. However, understanding a piece of text may require far more than just extracting the informa- tion from that piece itself. If the piece of text is a paragraph of a document, the reader may have to consider it together with other paragraphs in the document and the topic of the document. To un- derstand an utterance in a conversation, the utter- ance has to be put into the context of the conver- sation, which includes the goals of the participants and the dialog history. Hence the notion of context is an intrinsic component of language understand- ing. Inspired by recent works in dialog systems (Seo et al., 2017; Liu and Perez, 2017), we formal- ize the contextual sequence mapping problem as a sequence mapping problem with a strong con- trolling contextual element that regulates the flow of information. The system has two sources of signals: (i) the main text input, for example, the history utterance sequence in dialog systems or the sequence of words in language modelling; and (ii) the context signal, e.g., the previous utterance in a dialog system, the discourse information in contextual language modelling or the question in question answering. Our contribution in this work is two-fold. First, we propose a new family of recurrent unit, the Context-dependent Additive Recurrent Neural Network (CARNN), specifically constructed for contextual sequence mapping. Second, we de- sign novel neural network architectures based on CARNN for dialog systems and contextual lan- guage modelling, and enhance the state of the art architecture (IWAN (Shen et al., 2017)) on question answering. Our novel building block, the CARNN, draws inspiration from the Recur- rent …

Content truncated to 500 words