What Makes Reading Comprehension Questions Difficult?

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 628 - 647 May 22-27, 2022 c⃝2022 Association for Computational Linguistics AraT5: Text-to-Text Transformers for Arabic Language Generation El Moatez Billah Nagoudi⋆ AbdelRahim Elmadany⋆ Muhammad Abdul-Mageed⋆ Deep Learning and Natural Language Processing Group The University of British Columbia {moatez.nagoudi,a.elmadany,muhammad.mageed}@ubc.ca Abstract Transfer learning with a unified Transformer frame- work (T5) that converts all language problems into a text-to-text format was recently proposed as a simple and effective transfer learning approach. Although a multilingual version of the T5 model (mT5) was also introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialects–Arabic. For evaluation, we introduce a novel benchmark for ARabic language GENeration (ARGEN), covering seven important tasks. For model comparison, we pre-train three powerful Arabic T5-style models and evaluate them on ARGEN. Although pre-trained with ∼ 49% less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al., 2021). Our models are publicly available. We also link to individual ARGEN datasets through our public repository.1 1 Introduction Due to their remarkable ability to transfer knowl- edge from unlabeled data to downstream tasks, pre-trained Transformer-based language models have emerged as important components of mod- ern natural language processing (NLP) systems. In particular, the unified framework that converts all text-based language problems into a text-to-text for- mat presented through the T5 model (Raffel et al., 2019) is attractive. In addition to its simplicity, this approach is effective since it allows knowledge transfer from high-resource to low-resource tasks 1https://github.com/UBC-NLP/araT5 ⋆ All authors contributed equally. Figure 1: Our AraT5 encoder-decoder model and prompt samples from four investigated tasks, namely: title generation, machine translation, question generation, and paraphrasing. without the need for changing model architecture. Unlike models such as BERT (Devlin et al., 2019), which are based on encoders only, the T5 model is an encoder-decoder that can naturally be em- ployed for natural language generation. Although the T5 model, originally pre-trained for English, was recently extended to the multilingual setting as mT5 (Xue et al., 2020), it is not clear how suited it is to individual languages (and varieties of these languages). In addition, systematic issues have been discovered in multilingual corpora on which language models have been trained (Kreutzer et al., 2021). In absence of comparisons with monolin- gual pre-trained language models that serve differ- ent non-English contexts, it remains unknown how multilingual models really fare against language- specific models. In this work, we offer the first comparison of the mT5 model to similar encoder-decoder models ded- icated to Arabic. We choose Arabic as our context due to its large set of diverse varieties as well as its wide use on …

Content truncated to 500 words