Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1462–1471 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1462 Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith Shatha Altammami12, Eric Atwell2 King Saud University1, University of Leeds2 Saudi Arabia, UK Shaltammami@ksu.edu.sa1 {Scshal, E.S.Atwell}@leeds.ac.uk2 Abstract Transformer-based models showed near-perfect results on several downstream tasks. However, their performance on classical Arabic texts is largely unexplored. To fill this gap, we evaluate monolingual, bilingual, and multilingual state-of-the-art models to detect relatedness between the Quran (Muslim holy book) and the Hadith (Prophet Muhammed teachings), which are complex classical Arabic texts with underlying meanings that require deep human understanding. To do this, we carefully built a dataset of Quran-verse and Hadith-teaching pairs by consulting sources of reputable religious experts. This study presents the methodology of creating the dataset, which we make available on our repository, and discusses the models’ performance that calls for the imminent need to explore avenues for improving the quality of these models to capture the semantics in such complex, low-resource texts. Keywords: Hadith, Quran, dataset, semantic similarity. 1. Introduction Recently, state-of-the-art (SOTA) transformer-based models showed unprecedented performance that surpasses human scores on common benchmarks like SICK (Marelli et al., 2014), which created a hype into believing such models could ‘understand’ textual data (Bender and Koller, 2020). However, an emerging trend of probing these models yields striking results that suggest models are not able to generalize, but rather are more likely to memorize (Elangovan et al., 2021). This is due to the common practice of generat- ing training and testing data using random split, which inadvertently leads to data leakage from the training set to the testing set reaching 70% of instance overlap (Lewis et al., 2020). Moreover, existing benchmark datasets possess a low readability index which does not reflect real-world complex data (Chandrasekaran and Mago, 2020). This is aggravated for classical Arabic (CA) since it is considered low-resource in terms of available datasets and inherently challenging for natural language processing (NLP) tasks (Habash, 2010). Equipped with these observations, we evaluate the performance of SOTA models on a binary classifi- cation task of identifying whether two pieces of CA text convey the same underlying message, which is a special case of semantic similarity. However, before undergoing the experiments, we introduce the datasets that address the two aforementioned limitations: low-readability and data leakage. To address the low-readability index limitation, we created a new CA dataset of related religious texts from the Quran (Muslim holy book) and the Hadith (Prophet Muhammed teachings). It comprises em- bedded meanings which require reasoning and deep human understanding. Moreover, these texts contain complex syntactic and rhetorical features including, verbal idioms, sarcasm, hyperbole, rhetorical questions just to name a few (Abdul-Raof, 2013). To mitigate the limitation of data-leakage, we fine-tune the models on our extended version of the Qursim (Sharaf and Atwell, 2012) dataset consisting of related Quran-verse pairs. Then we test the best performing models …

Content truncated to 500 words