The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1584–1594 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1584 The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild Taja Kuzman, Peter Rupnik, Nikola Ljubeˇsi´c Joˇzef Stefan Institute Jamova cesta 39, 1000 Ljubljana taja.kuzman@ijs.si, peter.rupnik@ijs.si, nikola.ljubesic@ijs.si Abstract This paper presents a new training dataset for automatic genre identification GINCO, which is based on 1,125 crawled Slovenian web documents that consist of 650,000 words. Each document was manually annotated for genre with a new annotation schema that builds upon existing schemata, having primarily clarity of labels and inter-annotator agreement in mind. The dataset consists of various challenges related to web-based data, such as machine translated content, encoding errors, multiple contents presented in one document etc., enabling evaluation of classifiers in realistic conditions. The initial machine learning experiments on the dataset show that (1) pre-Transformer models are drastically less able to model the phenomena, with macro F1 metrics ranging around 0.22, while Transformer-based models achieve scores of around 0.58, and (2) multilingual Transformer models work as well on the task as the monolingual models that were previously proven to be superior to multilingual models on standard NLP tasks. Keywords: automatic genre identification, web genres, genre classification schema, web corpora, Slovenian language 1. Introduction With the arrival of the Web, it has become significantly easier to collect very large corpora that fuel innovation and creation of advanced resources and language tech- nologies. However, contrary to the traditionally col- lected corpora, web corpora are built in an automated way which limits the control over the contents that con- stitute the final corpus (Baroni et al., 2009). One of the post hoc evaluation methods to investigate the cor- pus composition and quality, and to enrich the corpus with important metadata is automatic genre identifi- cation (AGI). This method focuses on genres as text categories based on the author’s purpose, the socially recognized function of a document and/or the conven- tional patterns of form, following the definition by Or- likowski and Yates (1994). As this research is a part of the MaCoCu1 project that aims to collect large corpora for under-resourced lan- guages, our main purpose is to provide a classifier that would efficiently identify genres in web corpora for Slovenian and other languages, which would allow an in-depth analysis of the quality and composition of the newly provided corpora. In addition to this, annotating the data with genre is beneficial in many other areas where language tech- nology can be improved by a more fine-grained docu- ment typology, such as in part-of-speech tagging (Gies- brecht and Evert, 2009), zero-shot dependency parsing (M¨uller-Eberstein et al., 2021), automatic summariza- tion (Stewart and Callan, 2009), and machine trans- lation (Van der Wees et al., 2018). Furthermore, nu- merous genre annotation studies have been conducted with the aim of improving the Information Retrieval (IR) tools (Stubbe and Ringlstetter, 2007; Zu Eissen 1https://macocu.eu/ and Stein, 2004; Vidulin …

Content truncated to 500 words