Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 555–565, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields Fei Liu Timothy Baldwin Trevor Cohn School of Computing and Information Systems The University of Melbourne Victoria, Australia fliu3@student.unimelb.edu.au tb@ldwin.net t.cohn@unimelb.edu.au Abstract Despite successful applications across a broad range of NLP tasks, conditional ran- dom fields (“CRFs”), in particular the linear-chain variant, are only able to model local features. While this has important benefits in terms of inference tractabil- ity, it limits the ability of the model to capture long-range dependencies between items. Attempts to extend CRFs to cap- ture long-range dependencies have largely come at the cost of computational com- plexity and approximate inference. In this work, we propose an extension to CRFs by integrating external memory, taking in- spiration from memory networks, thereby allowing CRFs to incorporate informa- tion far beyond neighbouring steps. Ex- periments across two tasks show substan- tial improvements over strong CRF and LSTM baselines. 1 Introduction While long-range contextual dependencies are prevalent in natural language, for tractability rea- sons, most statistical models capture only local features (Finkel et al., 2005). Take the sentence in Figure 1, for example. Here, while it is easy to determine that Interfax in the second sentence is a named entity, it is hard to determine its semantic class, as there is little context information. The us- age in the first sentence, on the other hand, can be reliably disambiguated due to the post-modifying phase news agency. Ideally we would like to be able to share such contexts across all usages (and variants) of a given named entity for reliable and consistent identification and disambiguation. A related example is forum thread discourse analysis. Previous work has largely focused on linear-chain Conditional Random Fields (CRFs) (Wang et al., 2011; Zhang et al., 2017), framing the task as one of sequence tagging. Although CRFs are adept at capturing local structure, the problem does not naturally suit a linear sequen- tial structure, i.e. , a post may be a reply to ei- ther a neighbouring post or one posted far earlier within the same thread. In both cases, contextual dependencies can be long-range, necessitating the ability to capture dependencies between arbitrarily distant items. Identifying this key limitation, Sut- ton and McCallum (2004) and Finkel et al. (2005) proposed the use of CRFs with skip connections to incorporate long-range dependencies. In both cases the graph structure must be supplied a pri- ori, rather than learned, and both techniques incur the need for costly approximate inference. Recurrent neural networks (RNNs) have been proposed as an alternative technique for encoding sequential inputs, however plain RNNs are unable to capture long-range dependencies (Bengio et al., 1994; Hochreiter et al., 2001) and variants such as LSTMs (Hochreiter and Schmidhuber, 1997), although more capabable of capturing non-local patterns, still exhibit a significant locality bias in practice (Lai et al., 2015; Linzen et al., 2016). In this …

Content truncated to 500 words