Is GPT-4 a Good Islamic Expert for Answering Quran Questions?

Is GPT-4 a Good Islamic Expert for Answering Quran Questions? Sarah Alnefaie King Abdulaziz University, Jeddah, Saudi Arabia University of Leeds, Leeds, UK scsaln@leeds.ac.uk Eric Atwell University of Leeds Leeds, UK e.s.atwell @leeds.ac.uk Mohammad Ammar Alsalka University of Leeds Leeds, UK m.a.alsalka @leeds.ac.uk Abstract In this research, we investigated GPT-4 as a question-answering model for the Holy Quran. As a first step, we built the Quran ques- tion–answer pair (QUQA) dataset, comprising 2,189 questions, and made it freely available via our repository. This dataset was then used to benchmark the performance of the current Generative Pre-trained Transformer 4 (GPT-4) model from the OpenAI research laboratory. The results show that GPT-4 did not do well with this dataset, with a 0.23 partial Average Precision (pAP) score, 0.26 F1@1 score, and 0.19 Exact Match (EM) score. Therefore, fur- ther improvement is needed for Classical Ara- bic responses generated by GPT model. Keywords: GPT-4, large language model, and Quran question–answer pair (QUQA) dataset 1 Introduction Vaswani et al. (2017) introduced the transformer architecture that significantly advanced the field of natural language processing (NLP). Bidirectional encoder representations from transformers (BERT) is one of the pre-trained transformer language mod- els that has outperformed the state of the art in many NLP downstream tasks, such as text classifi- cation, question answering and machine translation (Devlin et al., 2018). A challenging task in NLP is answering ques- tions in Arabic, especially from Classical Arabic texts, such as the Quran and Hadith Sharif. The challenge lies in the nature and features of the text. For example, the meaning and spelling of the terminology in the Quran differ from other Arabic variants (colloquial Arabic, Modern Stan- dard Arabic and Classical Arabic) (Altammami and Atwell, 2022). This challenge has motivated sev- eral researchers to use the BERT transformer for a Holy Quran question-answering system, and it has shown promising results (Ahmed et al., 2022; Alsaleh et al., 2022; ElKomy and Sarhan, 2022; Malhas and Elsayed, 2022) Recently, OpenAI has developed and provided access to several versions of the large language model (LLM)-driven ChatGPT—“GPT” refers to its underlying generative pre-trained transformer model (Brown et al., 2020; Ouyang et al., 2022). Several studies have focused on testing GPT on downstream tasks (Jiao et al., 2023; Wang et al., 2023; Qin et al., 2023). However, to the best of our knowledge, there has not yet been a published examination of how well GPT works in answering Islamic questions. Our research sought to answer the question, “Is GPT a good Islamic expert?” To answer this ques- tion, we first created a corpus for Islamic questions and answered them using the Noble Quran due to the lack of an available dataset of this type. We then tested the performance of GPT with these questions to learn about its behaviour, strengths and disad- vantages. The results of this study will benefit a large segment of Muslims worldwide, motivate additional research to improve upon any defects found and help researchers choose an appropriate language model in the …

Content truncated to 500 words