Handling Extreme Class Imbalance in Technical Logbook Datasets

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4034–4045 August 1–6, 2021. ©2021 Association for Computational Linguistics 4034 Handling Extreme Class Imbalance in Technical Logbook Datasets Farhad Akhbardeh, Cecilia Ovesdotter Alm, Marcos Zampieri, Travis Desell Rochester Institute of Technology Rochester, NY, USA {fa3019, coagla, mazgla, tjdvse}@rit.edu Abstract Technical logbooks are a challenging and under-explored text type in automated event identification. These texts are typically short and written in non-standard yet technical lan- guage, posing challenges to off-the-shelf NLP pipelines. The granularity of issue types de- scribed in these datasets additionally leads to class imbalance, making it challenging for models to accurately predict which issue each logbook entry describes. In this paper we fo- cus on the problem of technical issue classi- fication by considering logbook datasets from the automotive, aviation, and facilities mainte- nance domains. We adapt a feedback strategy from computer vision for handling extreme class imbalance, which resamples the training data based on its error in the prediction process. Our experiments show that with statistical sig- nificance this feedback strategy provides the best results for four different neural network models trained across a suite of seven different technical logbook datasets from distinct tech- nical domains. The feedback strategy is also generic and could be applied to any learning problem with substantial class imbalances. 1 Introduction Predictive maintenance techniques are applied to engineering systems to estimate when maintenance should be performed to reduce costs and improve operational efficiency (Carvalho et al., 2019), as well as mitigate risk and increase safety. Mainte- nance records are an important source of informa- tion for predictive maintenance (McArthur et al., 2018). These records are often stored in the form of technical logbooks in which each entry contains fields that identify and describe a maintenance issue (Akhbardeh et al., 2020a). Being able to classify these technical events is an important step in the development of predictive maintenance systems. In most technical logbooks, issues are manually labeled by domain experts (e.g., mechanics) in free text fields. This text can then be used to classify or cluster events by semantic similarity. Classifying events in technical logbooks is a challenging prob- lem for the NLP community for several reasons: (a) the technical logbooks are written by various do- main experts and contain short text entries with non- standard language including domain-specific ab- breviated words (see Table 1 for examples), which makes them distinct from other short non-standard text corpora (e.g., social media); (b) off-the-shelf NLP tools struggle to perform well on this type of data as they tend to be trained on standard contem- porary corpora such as newspaper texts; (c) outside of the clinical and biomedical sciences, there is a lack of domain-specific, expert-based datasets for studying expert-based event classification, and in particular few resources are available for techni- cal problem domains; and (d) technical logbooks tend to be characterized by a large number of event classes that are highly imbalanced. …

Content truncated to 500 words