Capturing Pragmatic Knowledge in Article Usage Prediction using LSTMs

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2625–2634, Osaka, Japan, December 11-17 2016. Capturing Pragmatic Knowledge in Article Usage Prediction using LSTMs Jad Kabbara Yulan Feng School of Computer Science McGill University Montreal, QC, Canada {jad@cs, yulan.feng@mail, jcheung@cs}.mcgill.ca Jackie Chi Kit Cheung Abstract We examine the potential of recurrent neural networks for handling pragmatic inferences in- volving complex contextual cues for the task of article usage prediction. We train and compare several variants of Long Short-Term Memory (LSTM) networks with an attention mechanism. Our model outperforms a previous state-of-the-art system, achieving up to 96.63% accuracy on the WSJ/PTB corpus. In addition, we perform a series of analyses to understand the impact of various model choices. We find that the gain in performance can be attributed to the ability of LSTMs to pick up on contextual cues, both local and further away in distance, and that the model is able to solve cases involving reasoning about coreference and synonymy. We also show how the attention mechanism contributes to the interpretability of the model’s effectiveness. 1 Introduction Correctly performing pragmatic reasoning is at the core of many NLP tasks such as information ex- traction, automatic summarization, and machine translation. We focus in this paper on definiteness prediction, the task of determining whether a noun phrase should be definite or indefinite. In English, one instantiation of this task is to predict whether to use a definite article (the), indefinite article (a(n)), or no article at all. It has applications in machine translation (Heine, 1998; Netzer and Elhadad, 1998), and in L2 grammatical error detection and correction (Han et al., 2006). Definiteness prediction is an interesting testbed for pragmatic reasoning, because both contextual and local cues are crucial to determining the acceptability of a particular choice of article. Consider the following example: (1) A/#the man entered the room. The/#a man turned on the TV. Factors such as discourse context, familiarity, and information status play a role in determining the choice of articles. Here, man is introduced into the discourse context by an indefinite article, then subsequently referred to by a definite article. On the other hand, non-context-dependent factors such as local syntactic and semantic restrictions may block the presence of an article. For example, demonstratives (e.g., this, that), certain quantifiers (e.g., no), and mass nouns (e.g., money) do not permit articles. In this work, we investigate Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhu- ber, 1997), a subclass of recurrent neural networks (RNNs) which have been popular recently in a variety of NLP tasks (Sutskever et al., 2014; Mikolov et al., 2010; Dyer et al., 2015). A number of reasons are often cited for the impressive performance gains of RNNs (Goldberg, 2015). First, they are able to take advantage of the patterns inherent in the data to learn features and representations suitable for complex interpretation tasks. Second, they can learn connections between processing units in the same layer, al- lowing the network to capture relations and patterns …

Content truncated to 500 words