Estimating Confidence of Predictions of Individual Classifiers and TheirEnsembles for the Genre Classification Task

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5974–5982 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5974 Estimating Confidence of Predictions of Individual Classifiers and Their Ensembles for the Genre Classification Task Mikhail Lepekhin, Serge Sharoff MIPT, University of Leeds Moscow, UK lepehin.mn@phystech.edu, S.Sharoff@leeds.ac.uk Abstract Genre identification is a kind of non-topic text classification. The main difference between this task and topic classification is that genres, unlike topics, usually cannot be expressed just by simple keywords, and thus they need to be defined in terms of their functions in communication. Neural models based on pre-trained transformers, such as BERT or XLM-RoBERTa, demonstrate SOTA results in many NLP tasks, including non-topical classification. However, in many cases, their downstream application to very large corpora, such as those extracted from social media, can lead to unreliable results because of dataset shifts, when some raw texts do not match the profile of the training set. To mitigate this problem, we experiment with individual models as well as with their ensembles. To evaluate the robustness of all models we use a prediction confidence metric, which estimates the reliability of a prediction in the absence of a gold standard label. We can evaluate robustness via the confidence gap between the correctly classified texts and the misclassified ones on a labeled test corpus, higher gaps make it easier to improve our confidence that our classifier made the right decision. Our results show that for all of the classifiers tested in this study, there is a confidence gap, but for the ensembles, the gap is wider, meaning that ensembles are more robust than their individual models. Keywords: Document Classification, Evaluation Methodologies, Language Modeling, pre-trained transformers 1. Introduction Non-topical text classification includes a wide range of tasks aimed at predicting a text property that is not con- nected directly to a text topic. For example, predicting a text style, politeness, difficulty level, the age or the first language of its author, etc. Automatic genre iden- tification (Santini et al., 2010) is one of the standard problems of non-topical text classification. It is applied in many areas such as information retrieval, language teaching or linguistic research. Compared to topical text classification, genre classifi- cation has additional difficulties. First of all, the con- cept of genre is more complex than that of a topic. Ev- ery topic has a set of keywords and, therefore, one can define whether a text belongs to a topic or not based on the occurrence of the keywords in the text. Genre is a functional dimension of texts and in most cases, it cannot be defined just by a set of its keywords. Be- sides, genre typologies are often very large, while the training datasets offer few examples per more specific genres, which makes the genre harder to be classified by neural models. Text genre identification has a long history of research. (Sharoff et al., 2010) is one of the first works on text genre …

Content truncated to 500 words