Automatic dialect classification for statistical machine translation

Automatic Dialect Classification for Statistical Machine Translation Saab Mansour mansour@cs.rwth-aachen.de Aachen University, Aachen, Germany Yaser Al-Onaizan onaizan@us.ibm.com Graeme Blackwood blackwood@us.ibm.com Christoph Tillmann ctill@us.ibm.com IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Abstract The training data for statistical machine translation are gathered from various sources represent- ing a mixture of domains. In this work, we argue that when translating dialects representing varieties of the same language, a manually assigned data source is not a reliable indicator of the dialect. We resort to automatic dialect classification to refine the training corpora according to the different dialects and build improved dialect specific systems. A fairly standard classifier for Arabic developed within this work achieves state-of-the-art performance, with classification precision above 90%, making it usefully accurate for our application. The classification of the data is then used to distinguish between the different dialects, split the data accordingly, and utilize the new splits for several adaptation techniques. Performing translation experiments on a large scale dialectal Arabic to English translation task, our results show that the classifier gen- erates better contrast between the dialects and achieves superior translation quality than using the original manual corpora splits. 1 Introduction Training data for statistical machine translation (SMT) are extracted from various sources repre- senting different domains (e.g., newswire, webforums, ...). The source of the data (encapsulated by meta-information) can be utilized to perform domain adaptation using different techniques. For example, mixture modeling of grammars trained on different sources of data (Foster and Kuhn, 2007), or provenance features using different sources of data (Chiang et al., 2011). The meta-information based corpora split may contain further domain granularities. In this work, we tackle the case where the corpora contain a mixture of dialects. Dialects refer to varieties of a language, differing by vocabulary, morphology, grammar, etc. In this scenario, the meta-information split is rendered unreliable, and better splitting is required to achieve improvements using standard adaptation methods. Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 342 We start with developing an automatic dialect classifier for the purpose of refining the corpora splits. The classifier is applied on an Arabic dialect identification task, where we distin- guish between the Egyptian Arabic (ARZ) and Modern Standard Arabic (MSA) dialects. MSA is the standard written form of Arabic while ARZ and other dialectal forms are mainly used for speech. Due to the prevalence of MSA in written form, most of the corpora collected for training SMT systems contain a majority of MSA data. Nevertheless, dialectal data has a strong presence in on-line content such as weblogs, forums and user commentary. Applying an auto- matic dialect classifier on corpora designated as dialectal shows that a large portion of the data is actually MSA, making dialectal identification essential for a successful utilization of the data. Next, we extensively experiment with applying the classifier output for domain-adaptation, and compare using the classifier output to using meta-information based data splits. Various adaptation methods are investigated, including: domain-specific …

Content truncated to 500 words