SoMeWeTa: A Part-of-Speech Tagger for German Social Media and Web Texts

SoMeWeTa: A Part-of-Speech Tagger for German Social Media and Web Texts Thomas Proisl Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg Professur f¨ur Korpuslinguistik Bismarckstr. 6, 91054 Erlangen, Germany thomas.proisl@fau.de Abstract Off-the-shelf part-of-speech taggers typically perform relatively poorly on web and social media texts since those domains are quite different from the newspaper articles on which most tagger models are trained. In this paper, we describe SoMeWeTa, a part-of-speech tagger based on the averaged structured perceptron that is capable of domain adaptation and that can use various external resources. We train the tagger on the German web and social media data of the EmpiriST 2015 shared task. Using the TIGER corpus as background data and adding external information about word classes and Brown clusters, we substantially improve on the state of the art for both the web and the social media data sets. The tagger is available as free software. Keywords: part-of-speech tagging, domain adaptation, evaluation 1. Introduction Part-of-speech tagging is a core task in natural language processing (NLP) that is important for many subsequent processing steps, e. g. lemmatization, parsing, named-entity recognition or machine translation. There is already a large number of part-of-speech taggers available that provide pre-trained models for many different languages, including German. Those models are almost always trained on corpora of edited texts written by profes- sional writers, usually newspaper articles. This is also true for German models that are usually trained on the TIGER corpus (Brants et al., 2004). Unfortunately, models trained on newspaper articles perform relatively poorly on web and social media data (Giesbrecht and Evert, 2009). This is largely due to the many unconventional spelling variants that occur in web and social media texts and that re- sult in a high proportion of out-of-vocabulary (OOV) words on which most taggers perform much worse than on in- vocabulary words. In addition, there are several phenomena in web and social media texts that usually do not occur in edited texts and that cannot be captured properly by most part-of-speech tagsets, including STTS (Schiller et al., 1999). Those phenomena include emoticons, interaction words (e. g. *lach*), hash tags, addressing terms, URLs, onomatopoeia, and colloquial contractions (e. g. machste). One of the aims of the EmpiriST 2015 shared task (Beißwenger et al., 2016)1 was to improve tokenization and part-of-speech tagging of German computer-mediated communication (CMC) and web corpora. To this end, a gold standard of more than 22,000 tokens (the precise num- bers are summarized in Table 1) was annotated with an extended version of STTS called STTS IBK (Beißwenger et al., 2015). STTS IBK introduces 18 additional tags, mainly for phenomena typically found in CMC data. In the present paper, we describe SoMeWeTa (short for Social Media and Web Tagger),2 a freely available part-of- 1https://sites.google.com/site/ empirist2015/ 2https://github.com/tsproisl/SoMeWeTa CMC Web Training 5,109 4,944 Test 5,234 7,568 Total 10,343 12,512 Table 1: Sizes of the EmpiriST training and test sets in tokens. speech tagger that achieves the best results on the EmpiriST 2015 data published so far. 2. Related Work Four teams participated in …

Content truncated to 500 words