To Case or not to case: Evaluating Casing Methods for Neural Machine Translation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3752–3760 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3752 To Case or not to case: Evaluating Casing Methods for Neural Machine Translation Thierry Etchegoyhen and Harritxu Gete Ugarte Department of Speech and Natural Language Technologies, Vicomtech Mikeletegi Pasalekua, 57, Donostia, Gipuzkoa, Spain {tetchegoyhen, hgete}@vicomtech.org Abstract We present a comparative evaluation of casing methods for Neural Machine Translation, to help establish an optimal pre- and post-processing methodology. We trained and compared system variants on data prepared with the main casing methods available, namely translation of raw data without case normalisation, lowercasing with recasing, truecasing, case factors and inline casing. Machine translation models were prepared on WMT 2017 English-German and English-Turkish datasets, for all translation directions, and the evaluation includes reference metric results as well as a targeted analysis of case preservation accuracy. Inline casing, where case information is marked along lowercased words in the training data, proved to be the optimal approach overall in these experiments. Keywords: Neural Machine Translation, Casing, Evaluation 1. Introduction Neural Machine Translation (NMT) (Cho et al., 2014; Bah- danau et al., 2015; Vaswani et al., 2017) has become the dominant paradigm of machine translation (MT) research and development in recent years. As a data-driven method, where translation knowledge is induced from multilingual corpora, NMT modelling typically involves preprocessing steps that notably include tokenisation, case normalisation and word segmentation into subword units, as demonstrated for instance by the system descriptions in the annual WMT translation shared tasks (see, e.g., (Bojar et al., 2017) and references therein). Whereas different approaches to tokenisation and subword generation have been proposed and evaluated in different studies, the optimal handling of casing has not been sys- tematically evaluated. In this work, we evaluate the main methods to handle capitalisation, namely training without case normalisation, lowercasing with recasing, truecasing, case factors and inline casing. The latter method, which is based on inserting case tags along lowercased words and does not require any extension of NMT infrastructures is shown to obtain the best results overall. The different approaches to casing are evaluated on two standard datasets, namely WMT 2017 for German-English and Turkish-English, in both translation directions, thus covering translation cases which differ in terms of capital- isation rules. Translation results are evaluated on the WMT test sets and on additional datasets that include newspaper titles, which feature extended capitalisation usage in En- glish, and subtitles, which involve specific uses of capitali- sation as well. Our main contribution is thus the first systematic compari- son between the main casing methods available for Neural Machine Translation, measuring the impact of the differ- ent variants and helping determine optimal data preparation pipelines for NMT system development. The remainder of the paper is organised as follows: Sec- tion 2. describes related work in data preprocessing for machine translation; Section 3. presents the different meth- ods to be evaluated in this work; Section 4. describes the experimental setup and …

Content truncated to 500 words