Semantic Tagging with Deep Residual Networks

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531–3541, Osaka, Japan, December 11-17 2016. Semantic Tagging with Deep Residual Networks Johannes Bjerva University of Groningen The Netherlands j.bjerva@rug.nl Barbara Plank University of Groningen The Netherlands b.plank@rug.nl Johan Bos University of Groningen The Netherlands johan.bos@rug.nl Abstract We propose a novel semantic tagging task, sem-tagging, tailored for the purpose of multilin- gual semantic parsing, and present the first tagger using deep residual networks (ResNets). Our tagger uses both word and character representations, and includes a novel residual bypass archi- tecture. We evaluate the tagset both intrinsically on the new task of semantic tagging, as well as on Part-of-Speech (POS) tagging. Our system, consisting of a ResNet and an auxiliary loss func- tion predicting our semantic tags, significantly outperforms prior results on English Universal Dependencies POS tagging (95.71% accuracy on UD v1.2 and 95.67% accuracy on UD v1.3). 1 Introduction A key issue in computational semantics is the transferability of semantic information across languages. Many semantic parsing systems depend on sources of information such as POS tags (Pradhan et al., 2004; Copestake et al., 2005; Bos, 2008; Butler, 2010; Berant and Liang, 2014). However, these tags are often customised for the language at hand (Marcus et al., 1993) or massively abstracted, such as the Universal Dependencies tagset (Nivre et al., 2016). Furthermore, POS tags are syntactically oriented, and therefore often contain both irrelevant and insufficient information for semantic analysis and deeper semantic processing. This means that, although POS tags are highly useful for many downstream tasks, they are unsuitable both for semantic parsing in general, and for tasks such as recognising textual entailment. We present a novel set of semantic labels tailored for the purpose of multilingual semantic parsing. This tagset (i) abstracts over POS and named entity types; (ii) fills gaps in semantic modelling by adding new categories (for instance for phenomena like negation, modality, and quantification); and (iii) gener- alises over specific languages (see Section 2). We introduce and motivate this new task in this paper, and refer to it as semantic tagging. Our experiments aim to answer the following two research questions: 1. Given an annotated corpus of semantic tags, it is straightforward to apply off-the-shelf sequence taggers. Can we significantly outperform these with recent neural network architectures? 2. Semantic tagging is essential for deep semantic parsing. Can we find evidence that semtags are effective also for other NLP tasks? To address the first question, we will look at convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are both highly prominent approaches in the recent natural language processing (NLP) literature. A recent development is the emergence of deep residual networks (ResNets), a building block for CNNs. ResNets consist of several stacked residual units, which can be thought of as a collection of convolutional layers coupled with a ‘shortcut’ which aids the propagation of the signal in a neural network. This allows for the construction of much deeper networks, since keeping a ‘clean’ …

Content truncated to 500 words