A Linguistically Motivated Test Suite to Semi-Automatically Evaluate German–English Machine Translation Output

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 79–88 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 79 ZAEBUC: An Annotated Arabic-English Bilingual Writer Corpus Nizar Habash† and David Palfreyman‡ †New York University Abu Dhabi, Abu Dhabi, UAE, ‡Zayed University, Abu Dhabi, UAE nizar.habash@nyu.edu, david.palfreyman@zu.ac.ae Abstract We present ZAEBUC, an annotated Arabic-English bilingual writer corpus comprising short essays by first-year university students at Zayed University in the United Arab Emirates. We describe and discuss the various guidelines and pipeline processes we followed to create the annotations and quality check them. The annotations include spelling and grammar correction, morphological tokenization, Part-of-Speech tagging, lemmatization, and Common European Framework of Reference (CEFR) ratings. All of the annotations are done on Arabic and English texts using consistent guidelines as much as possible, with tracked alignments among the different annotations, and to the original raw texts. For morpho- logical tokenization, POS tagging, and lemmatization, we use existing automatic annotation tools followed by manual correction. We also present various measurements and correlations with preliminary insights drawn from the data and annota- tions. The publicly available ZAEBUC corpus and its annotations are intended to be the stepping stones for additional annotations. Keywords: Annotated Corpus, Learner Corpus, CEFR, Arabic, English 1. Introduction Over half the world’s population are estimated to use more than one language every day (Grosjean, 2010); however, language corpora in general tend to focus on specific languages rather than on bilingual writers. Even research on ‘learner corpora’ of writing in English (or another language) tends to compare this writing with a corpus of writing by other, ‘native’ users of the same language. In this paper, we discuss the development of a new kind of corpus, which focuses on a large set of bilin- gual writers, and comprises samples of their writing in both languages. The Zayed University Arabic-English Bilingual Undergraduate Corpus (ZAEBUC)1 is not a ‘parallel corpus’ of texts with their translations. In- stead, ZAEBUC matches comparable texts in different languages written by the same writer on different occa- sions. The corpus comprises short essays written by 397 first-year university students at Zayed University (ZU) in the United Arab Emirates totaling 388 English essays (87.6K words) and 214 Arabic essays (33.3K words). We enrich the corpus with a number of layered an- notations: (a) anonymized meta-data indicating extra- linguistic features of the writers and texts; (b) manually corrected versions of the raw text; (c) automatic and manual annotations to identify morphological tokens, part-of-speech (POS), and lemmas; and (d) writing profi- ciency ratings using the Common European Framework of Reference (CEFR) (Council of Europe, 2001). The ZAEBUC dataset is an open, publicly available2 and extendable research resource, designed with the in- tention to support empirically driven research in Arabic, English and bilingual development, as well as research and system development in natural language processing (NLP). 1The Arabic word �������� zi’baq means ‘mercury’. 2http://www.zaebuc.org The paper is organized as follows: Section 2 presents some related work and …

Content truncated to 500 words