Normalization of Dutch User-Generated Content

Proceedings of Recent Advances in Natural Language Processing, pages 179–188, Hissar, Bulgaria, 7-13 September 2013. Normalization of Dutch User-Generated Content Orph´ee De Clercq1,2, Sarah Schulz3, Bart Desmet1,2, Els Lefever1,2 and V´eronique Hoste1,3 1 LT3, Language and Translation Technology Team – University College Ghent Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium 2 Department of Applied Mathematics and Computer Science – Ghent University Krijgslaan 281 (S9), 9000 Ghent, Belgium 3 Department of Lingusitics – Ghent University Blandijnberg 1, 9000 Ghent, Belgium Firstname.Lastname@UGent.be Abstract This paper describes a phrase-based ma- chine translation approach to normalize Dutch user-generated content (UGC). We compiled a corpus of three different so- cial media genres (text messages, message board posts and tweets) to have a sample of this recent domain. We describe the var- ious characteristics of this noisy text ma- terial and explain how it has been man- ually normalized using newly developed guidelines. For the automatic normaliza- tion task we focus on text messages, and find that a cascaded SMT system where a token-based module is followed by a trans- lation at the character level gives the best word error rate reduction. After these ini- tial experiments, we investigate the sys- tem’s robustness on the complete domain of UGC by testing it on the other two so- cial media genres, and find that the cas- caded approach performs best on these genres as well. To our knowledge, we deliver the first proof-of-concept system for Dutch UGC normalization, which can serve as a baseline for future work. 1 Introduction In the past two decades, many resources have been invested to develop state-of-the-art text processing tools for Dutch1. Similar to other reported lan- guages, these tools, which have all been devel- oped with standard text in mind, show a signif- icant drop in performance when applied to user- generated content (UGC). This is for example the 1Among others, in the framework of the STEVIN pro- gramme, see Spijns and Odijk (2013) for an overview. case when applying parsing (Foster et al., 2011) or named entity recognition (Liu et al., 2011b; Ritter et al., 2011) to Twitter data. Typical problems that hinder automatic text processing include the use and productivity of abbreviations, deliberate mis- spellings, phonetic text, colloquial and ungram- matical language use, lack of punctuation and in- consistent capitalization. No systems currently exist to automatically nor- malize Dutch noisy text into its standard equiva- lent. In order to develop a system which can han- dle different types of user-generated content, we collected and studied three social media genres: text messages, message board posts and tweets. In this paper, we investigate the viability of adopting a character-based machine translation approach to the normalization task. This is different from previous research investigating MT approaches for normalization, which has mainly focused on token-based translation (Aw et al., 2006; Kobus et al., 2008). For our experiments we first focus on the genre that poses the largest number of normalization challenges in our corpus, namely text messages, in order to have a proof of concept. …

Content truncated to 500 words