The SADID Evaluation Datasets for Low-Resource Spoken Language Machine Translation of Arabic Dialects

Proceedings of the 28th International Conference on Computational Linguistics, pages 6030–6043 Barcelona, Spain (Online), December 8-13, 2020 6030 The SADID Evaluation Datasets for Low-Resource Spoken Language Machine Translation of Arabic Dialects Wael Abid Department of Computer Science Stanford University wael@cs.stanford.edu Abstract Low-resource Machine Translation recently gained a lot of popularity, and for certain languages, it has made great strides. However, it is still difficult to track progress in other languages for which there is no publicly available evaluation data. In this paper, we introduce benchmark datasets for Arabic and its dialects. We describe our design process and motivations and analyze the datasets to understand their resulting properties. Numerous successful attempts use large monolingual corpora to augment low-resource pairs. We try to approach augmentation differently and investigate whether it is possible to improve MT models without any external sources of data. We accomplish this by bootstrapping existing parallel sentences and complement this with multilingual training to achieve strong baselines. 1 Introduction Machine Translation (MT) models have achieved state-of-the-art results on several high-resource lan- guage pairs. This success is mostly due to advances in modeling and the availability of clean parallel corpora. Yet, advances in MT on low-resource language pairs are still lagging behind mainly due to the scarcity of parallel training data. Augmenting with back-translated large monolingual corpora (Sennrich et al., 2016a) can partly offset the effects of small amounts of training data, but it is not applicable for pairs where the target language doesn’t have large monolingual corpora. On the other hand, freely avail- able benchmark datasets across different tasks have historically provided reference points for researchers to drive their fields forward. Notable examples include The General Language Understanding Evaluation (GLUE) (Wang et al., 2018) for NLU and the Conference on Machine Translation (WMT) datasets for MT. Still, benchmark datasets for most low-resource language pairs remain a much-needed resource. Arabic dialects, like most low-resource languages, lack freely available benchmark datasets that can be used to evaluate models. This makes previous research results difficult to track and reproduce. In this paper, we introduce benchmark datasets for evaluation on MT tasks between Arabic dialects, Modern Standard Arabic (MSA) and English. We describe the design considerations and data collection guidelines we adopt. We provide an analysis and an empirical evaluation of state-of-the-art MT models on our benchmark datasets. We explore optimal unsupervised segmentation parameters and introduce a novel data augmentation method which we call bootstrapping. The goal of the project is to provide datasets that are reliable public evaluation benchmarks to track progress in the translation quality across different dialects. We put a specific emphasis on creating datasets that are not only able to assess the performance of MT systems but also test their robustness on two levels: the domain, and the dialectal diversity. 2 Related Work Machine Translation resources for Arabic are mostly focused on MSA. Nonetheless, there are a num- ber of efforts dedicated to dialects. The BOLT (Broad Operational Language Translation)1 program This work is licensed under a Creative Commons …

Content truncated to 500 words