Predicting the Level of Text Standardness in User-generated Content

Proceedings of Recent Advances in Natural Language Processing, pages 371–378, Hissar, Bulgaria, Sep 7–9 2015. Predicting the Level of Text Standardness in User-generated Content Nikola Ljubeši´c∗‡ Darja Fišer† Tomaž Erjavec∗ Jaka ˇCibej† Dafne Marko† Senja Pollak∗ Iza Škrjanec† ∗ Dept. of Knowledge Technologies, Jožef Stefan Institute name.surname@ijs.si † Dept. of Translation studies, Faculty of Arts, University of Ljubljana name.surname@ff.uni-lj.si ‡ Dept. of Inf. Sciences, Faculty of Humanities and Social Sciences, University of Zagreb Abstract Non-standard language as it appears in user-generated content has recently at- tracted much attention. This paper pro- poses that non-standardness comes in two basic varieties, technical and linguistic, and develops a machine-learning method to discriminate between standard and non- standard texts in these two dimensions. We describe the manual annotation of a dataset of Slovene user-generated content and the features used to build our re- gression models. We evaluate and dis- cuss the results, where the mean abso- lute error of the best performing method on a three-point scale is 0.38 for tech- nical and 0.42 for linguistic standard- ness prediction. Even when using no language-dependent information sources, our predictor still outperforms an OOV- ratio baseline by a wide margin. In addi- tion, we show that very little manually an- notated training data is required to perform good prediction. Predicting standardness can help decide when to attempt to nor- malise the data to achieve better annota- tion results with standard tools, and pro- vide linguists who are interested in non- standard language with a simple way of selecting only such texts for their research. 1 Introduction User-generated content (UGC) is becoming an in- creasingly frequent and important source of hu- man knowledge and people’s opinions (Crystal, 2011). Language use in social media is char- acterised by special technical and social circum- stances, and as such deviates from the norm of traditional text production. Researching the lan- guage of social media is not only of great value to (socio)linguists, but also beneficial for improving automatic processing of UGC, which has proven to be quite difficult (Sproat, 2001). Consistent de- creases in performance on noisy texts have been recorded in the entire text processing chain, from PoS-tagging, where the state-of-the-art Stanford tagger achieves 97% accuracy on Wall Street Jour- nal texts, but only 85% accuracy on Twitter data (Gimpel et al., 2011), to parsing, where double- digit decreases in accuracy have been recorded for 4 state-of-the-art parsers on social media texts (Petrov and McDonald, 2012). Non-standard linguistic features have been analysed both qualitatively and quantitatively (Eisenstein, 2013; Hu et al., 2013; Baldwin et al., 2013) and they have been taken into account in automatic text processing applications, which ei- ther strive to normalise non-standard features be- fore submitting them to standard text processing tools (Han et al., 2012), adapt standard process- ing tools to work on non-standard data (Gimpel et al., 2011) or, in task-oriented applications, use a series of simple pre-processing steps to tackle the most frequent UGC-specific phenomena (Foster et al., 2011). However, to the best …

Content truncated to 500 words