Exploring Different Dimensions of Attention for Uncertainty Detection

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 22–34, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Exploring Different Dimensions of Attention for Uncertainty Detection Heike Adel and Hinrich Sch¨utze Center for Information and Language Processing (CIS) LMU Munich, Germany heike@cis.lmu.de Abstract Neural networks with attention have proven effective for many natural lan- guage processing tasks. In this paper, we develop attention mechanisms for uncer- tainty detection. In particular, we gen- eralize standardly used attention mecha- nisms by introducing external attention and sequence-preserving attention. These novel architectures differ from standard approaches in that they use external re- sources to compute attention weights and preserve sequence information. We com- pare them to other configurations along different dimensions of attention. Our novel architectures set the new state of the art on a Wikipedia benchmark dataset and perform similar to the state-of-the-art model on a biomedical benchmark which uses a large set of linguistic features. 1 Introduction For many natural language processing (NLP) tasks, it is essential to distinguish uncertain (non- factual) from certain (factual) information. Such tasks include information extraction, question an- swering, medical information retrieval, opinion detection, sentiment analysis (Karttunen and Za- enen, 2005; Vincze, 2014a; D´ıaz et al., 2016) and knowledge base population (KBP). In KBP, we need to distinguish, e.g., “X may be Basque” and “X was rumored to be Basque” (uncertain) from “X is Basque” (certain) to decide whether to add the fact “Basque(X)” to a knowledge base. In this paper, we use the term uncertain infor- mation to refer to speculation, opinion, vague- ness and ambiguity. We focus our experiments on the uncertainty detection (UD) dataset from the CoNLL2010 hedge cue detection task (Farkas et al., 2010). It consists of two medium-sized corpora from different domains (Wikipedia and biomedical) that allow us to run a large number of comparative experiments with different neural networks and exhaustively investigate different di- mensions of attention. Convolutional and recurrent neural networks (CNNs and RNNs) perform well on many NLP tasks (Collobert et al., 2011; Kalchbrenner et al., 2014; Zeng et al., 2014; Zhang and Wang, 2015). CNNs are most often used with pooling. More re- cently, attention mechanisms have been success- fully integrated into CNNs and RNNs (Bahdanau et al., 2015; Rush et al., 2015; Hermann et al., 2015; Rockt¨aschel et al., 2016; Yang et al., 2016; He and Golub, 2016; Yin et al., 2016). Both pool- ing and attention can be thought of as selection mechanisms that help the network focus on the most relevant parts of a layer, either an input or a hidden layer. This is especially beneficial for long input sequences, e.g., long sentences or en- tire documents. We apply CNNs and RNNs to un- certainty detection and compare them to a number of baselines. We show that attention-based CNNs and RNNs are effective for uncertainty detection. On a Wikipedia benchmark, we improve the state of the art by more than 3.5 F1 …

Content truncated to 500 words