A Taxonomy for In-depth Evaluation of Normalization for User Generated Content

A Taxonomy for In-depth Evaluation of Normalization for User Generated Content Rob van der Goot, Rik van Noord, Gertjan van Noord University of Groningen Broerstraat 5, Groningen, r.m.van.der.goot@rug.nl, r.i.k.van.noord@rug.nl, g.j.m.van.noord@rug.nl Abstract In this work we present a taxonomy of error categories for lexical normalization, which is the task of translating user generated content to canonical language. We annotate a recent normalization dataset to test the practical use of the taxonomy and read a near-perfect agreement. This annotated dataset is then used to evaluate how an existing normalization model performs on the different categories of the taxonomy. The results of this evaluation reveal that some of the problematic categories only include minor transformations, whereas most regular transformations are solved quite well. Keywords: normalization, user generated content, social media, error taxonomy 1. Introduction For other natural language processing tasks, such as gram- matical error correction and machine translation, there al- ready exist detailed error taxonomies, which help in get- ting insights in the strengths and weaknesses of systems. For normalization, such an evaluation does not exist yet. Reynaert (2008) proposed an evaluation framework which evaluates the different sub-tasks in more detail; enabling the evaluation of error detection, candidate generation and candidate ranking. In the more recent shared task on lexical normalization hosted at the WNUT workshop (Baldwin et al., 2015b), the outputs of systems were evaluated on preci- sion, recall and F1 score. Orthogonal to these approaches, we propose a more in-depth evaluation of normalization, focusing on categories of different normalization replace- ments. Existing error taxonomies are unfortunately not suitable for the task of normalization, since the categories are substan- tially different. For machine translation, taxonomies as the Multidimensional Quality Metrics (Mariana, 2014) are pro- posed, which contains 3 main categories: accuracy, verity and fluency. The last category would be the most relevant for normalization, but the normalization task compromises a different variety of errors and anomalies. In grammatical error correction, often a more detailed taxonomy for errors is used; the default benchmark has 28 categories (Ng et al., 2014). However, many of the errors in this taxonomy are not annotated in the normalization benchmarks and many normalization replacements are not included in this taxon- omy. Different benchmarks for normalization specify the task slightly different; one striking example is the inclu- sion of the expansions of phrasal abbreviations (e.g. ‘lol’�→‘laughing out loud’). However, this might not be the desired output, since one might argue that this expansion does not represent the intended meaning. This reveals an- other potential use for a taxonomy of normalization actions: it enables us to filter the categories before training, and thus learn a model which only handles the desired categories. 2. Normalization There is ample of previous work on normalization, but there is no consensus about the scope of the normalization task. Some existing corpora with normalization annotation for English are shown in Table 1. The corpora by Baldwin and Li (2015) and Kaljahi et al. (2015) are also annotated with error categories. However, the guidelines for …

Content truncated to 500 words