The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions

The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions Joachim Daiber, Rob van der Goot ILLC University of Amsterdam, CLCG University of Groningen J.Daiber@uva.nl, R.van.der.Goot@rug.nl Abstract We introduce the Denoised Web Treebank: a treebank including a normalization layer and a corresponding evaluation metric for dependency parsing of noisy text, such as Tweets. This benchmark enables the evaluation of parser robustness as well as text normalization methods, including normalization as machine translation and unsupervised lexical normalization, directly on syntactic trees. Experiments show that text normalization together with a combination of domain-specific and generic part-of-speech taggers can lead to a significant improvement in parsing accuracy on this test set. Keywords: Parsing, Part-of-Speech Tagging, Social Media Processing, Web Treebank 1. Introduction The quality of automatic syntactic analysis of clean, in- domain text has improved steadily in recent decades. Out- of-domain text and grammatically noisy text, on the other hand, remain an obstacle and often lead to significant de- creases in parsing accuracy. Recently, a lot of effort has been put into adapting natural language processing tools, such as named entity recognition (Liu et al., 2012) and POS tagging (Gimpel et al., 2011), to noisy content. In this pa- per, we focus on dependency parsing of noisy text. Specif- ically, we are interested in how much parse quality can be gained by text normalization. For this, we introduce a new dependency treebank with a normalization layer. This new dataset can be used to quantify the influence of text normal- ization on the parsing of user-generated content. The contributions of this paper are as follows: (1) We in- troduce the Denoised Web Treebank, a new Twitter depen- dency treebank with a normalization layer; (2) we propose a corresponding noise-aware evaluation metric; and (3) we use this dataset and the metric as a benchmark to evaluate the impact of text normalization on dependency parsing of user-generated content. 2. Related Work For the domain of web data, various datasets and treebanks have been introduced. Table 1 provides an overview of all relevant English treebanks. The constituency treebanks mentioned here were cre- ated using the English Web Treebank annotation guide- lines (Bies et al., 2012), which are an addendum to the Penn Treebank guidelines (Bies et al., 1995). These guidelines discuss domain-specific phenomena, including adaptations of existing labels as well as the addition of new labels for novel linguistic constructions. Foster et al. (2011a) de- scribe a constituency treebank consisting of two domains; Twitter and sports forums. The Twitter part is of compa- rable size to our treebank and is described in more detail in Foster et al. (2011b). The dependency treebanks show greater diversity in anno- tation. The English Web Treebank (Silveira et al., 2014) is annotated using the Universal Dependencies guidelines with additional relation types for the web domain. A very different approach is taken for the annotation of the Twee- bank (Kong et al., 2014). In its format, individual words can be skipped in the annotation. This is motivated by the idea that …

Content truncated to 500 words