A Fine-Grained Analysis of BERTScore

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 110–116 November 10–11, 2021. ©2021 Association for Computational Linguistics 110 Tune In: The AFRL WMT21 News-Translation Systems Grant Erdmann, Jeremy Gwinnup, Timothy Anderson Air Force Research Laboratory {grant.erdmann, jeremy.gwinnup.1, timothy.anderson.20}@us.af.mil Abstract This paper describes the Air Force Research Laboratory (AFRL) machine translation sys- tems and the improvements that were devel- oped during the WMT21 evaluation campaign. This year, we explore various methods of adapting our baseline models from WMT20 and again measure improvements in perfor- mance on the Russian–English language pair. 1 Introduction As part of the 2021 Conference on Machine Trans- lation (wmt, 2021) news-translation shared task, the AFRL human language technology team partic- ipated in the Russian–English portion of the compe- tition. We experiment with OpenNMT-tf 1 (Klein et al., 2018) and Marian 2 (Junczys-Dowmunt et al., 2018) transformer (Vaswani et al., 2017) models trained as part of our WMT20 (Gwinnup and An- derson, 2020) efforts and apply varying continued- training and fine-tuning approaches (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016), in- cluding a new method to select a fine-tuning set from a separate, larger corpus not used in training. We submit an OpenNMT-based transformer sys- tem fine-tuned on newstest test sets from 2014- 2017 as our primary entry, and a Marian-based transformer system fine-tuned on newstest test sets from 2014-2018 as a contrast. 2 Data and Preprocessing Since most of our efforts focus on fine-tuning ex- isting models this year, we reuse the training cor- pus from our WMT20 systems which includes the following parallel corpora: Commoncrawl (Smith et al., 2013), Yandex3, UN v1.0 (Ziemski et al., tf/1Available at: https://github.com/OpenNMT/OpenNMT- 2Available at: https://github.com/marian-nmt/marian 3https://translate.yandex.ru/corpus? lang=en 2016), Paracrawl4(Esplà et al., 2019), Wikimatrix (Schwenk et al., 2019), and backtranslated data from our WMT17 system (Gwinnup et al., 2017) as well as Edinburgh’s WMT17 system (Sennrich et al., 2017) yielding a raw corpus of over 76.3 million lines. The new Russian–English version 8 Paracrawl corpus is reserved for tuning set selection as de- scribed in Section 2.3. 2.1 Data Preparation We re-use the fastText (Joulin et al., 2016b,a) based language ID filtered corpus with an ID threshold of 0.8 as described in Gwinnup and Anderson (2020), shown in Table 1, allowing us to make concrete progress comparisons to last year’s systems. 2.2 Data Augmentation with Speech Recognition-like output In order to build a larger pool of training data, we have created Automatic Speech Recognition (ASR) - like training data for the Russian–English transla- tion task. Whereas written text can include upper and lowercase characters, punctuation, special sym- bols, and numbers written using digits, transcripts produced by ASR systems are typically uncased with no punctuation, no special symbols, and num- bers written as spoken (e.g., 4.1% rendered as “four point one percent”). In previous experiments on an English-German spoken language translation task (Ore et al., 2020), we found that we could get an improvement in BLEU score by formatting the MT training data such that the source language text matched …

Content truncated to 500 words