Automatic Error Type Annotation for Arabic

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 596–606 November 10–11, 2021. ©2021 Association for Computational Linguistics 596 Automatic Error Type Annotation for Arabic Riadh Belkebir and Nizar Habash Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi {rb4822,nizar.habash}@nyu.edu Abstract We present ARETA, an automatic error type annotation system for Modern Standard Ara- bic. We design ARETA to address Arabic’s morphological richness and orthographic am- biguity. We base our error taxonomy on the Arabic Learner Corpus (ALC) Error Tagset with some modifications. ARETA achieves a performance of 85.8% (micro average F1 score) on a manually annotated blind test por- tion of ALC. We also demonstrate ARETA’s usability by applying it to a number of sub- missions from the QALB 2014 shared task for Arabic grammatical error correction. The re- sulting analyses give helpful insights on the strengths and weaknesses of different submis- sions, which is more useful than the opaque M2 scoring metrics used in the shared task. ARETA employs a large Arabic morphological analyzer, but is completely unsupervised oth- erwise. We make ARETA publicly available. 1 Introduction There has been a lot of interest recently in Auto- matic Error Evaluation for many languages. Many specialized shared tasks in grammatical error cor- rection (GEC) and text normalization have used tools like M2 Scorer (Dahlmeier and Ng, 2012) and ERRANT (Bryant et al., 2017). In contrast with the opaque results of M2 Scorer based on extracted edits, ERRANT, designed primarily for English, allows for deep interpretation of GEC er- ror types since it gives more detailed explanations. Error type explainability is helpful for many NLP applications, including second language learning. Arabic is a morphologically rich and complex language with a high degree of ambiguity at the or- thographic, morphological, syntactic, lexical and semantic levels (Habash, 2010). Figure 1 presents a motivating example for the complexity of Ara- bic error type annotation (discussion in Section 2). Previous Arabic text correction shared tasks like QALB 2014 (Mohit et al., 2014) and QALB 2015 !ﺎﻟﺴ%ﺎرەأﺣﻤﺮ ,- . اﻟﺴ%ﺎرة اﻟﺤﻤﺮاء bAlsyArh ÂHmr fy AlsyArħ AlHmrA’ by-the-carfs redms in the-carfs the-redfs morph changes ortho change semantic change Figure 1: An example of aligned erroneous source and reference phrases with three different error types. (Rozovskaya et al., 2015) evaluated submissions using the M2 Scorer. Alfaifi (2015) proposed a taxonomy for Arabic error types and annotated the Arabic Learner Corpus (ALC) using it; but, he does not provide an error classification tool. In this paper, we present ARETA, a system for the extraction and annotation of error types in Ara- bic. ARETA is inspired by ERRANT, but ad- dresses the unique and complex challenges of Ara- bic. We base our error taxonomy on the ALC Er- ror Tagset (Alfaifi and Atwell, 2014; Alfaifi, 2015) with some modifications. ARETA reaches a micro average F1 score of 85.8% on an ALC blind test. We also demonstrate ARETA’s usability on a num- ber of submissions from QALB 2014 (Mohit et al., 2014) shared task. …

Content truncated to 500 words