Multi-Fact Correction in Abstractive Text Summarization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 82–89, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 82 Simulated Multiple Reference Training Improves Low-Resource Machine Translation Huda Khayrallah Brian Thompson Matt Post and Philipp Koehn Johns Hopkins University {huda, brian.thompson, phi}@jhu.edu, post@cs.jhu.edu Abstract Many valid translations exist for a given sen- tence, yet machine translation (MT) is trained with a single reference translation, exacerbat- ing data sparsity in low-resource settings. We introduce Simulated Multiple Reference Train- ing (SMRT), a novel MT training method that approximates the full space of possible transla- tions by sampling a paraphrase of the reference sentence from a paraphraser and training the MT model to predict the paraphraser’s distri- bution over possible tokens. We demonstrate the effectiveness of SMRT in low-resource set- tings when translating to English, with im- provements of 1.2 to 7.0 BLEU. We also find SMRT is complementary to back-translation. 1 Introduction Variability and expressiveness are core features of language, and they extend to translation as well. Dreyer and Marcu (2012) showed that naturally oc- curring sentences have billions of valid translations. Despite this variety, machine translation (MT) mod- els are optimized toward a single translation of each sentence in the training corpus. Training a high re- source MT model on millions of sentence pairs likely exposes it to similar sentences translated dif- ferent ways, but training a low-resource MT model with a single translation for each sentence (out of potentially billions) exacerbates data sparsity. We hypothesize that the discrepancy between linguistic diversity and standard single-reference training hinders MT performance. This was previ- ously impractical to address, since obtaining multi- ple human translations of training data is typically not feasible. However, recent neural sentential para- phrasers produce fluent, meaning-preserving En- glish paraphrases. We introduce a novel method that incorporates such a paraphraser directly in the training objective, and uses it to simulate the full space of translations. turtle turtle turtle turtle rabbit A tortoise turtle rabbit tortoise hare hare tortoise tortoise beat the the won against the lost to the The lost to the turtle was by beaten tortoise the was by beaten turtle a against a Figure 1: Some ........ possible............. paraphrases of the origi- nal reference, ‘The tortoise beat the hare,’ for the Dutch source sentence, ‘De schildpad versloeg de haas.’ A sampled path and some of the other tokens also considered in the training objective are highlighted. We demonstrate the effectiveness of our method on two corpora from the low-resource MATERIAL program, and on bitext from GlobalVoices. We release data & code: data.statmt.org/smrt 2 Method We propose Simulated Multiple Reference Training (SMRT), which uses a paraphraser to approximate the full space of possible translations, since explic- itly training on billions of possible translations per sentence is intractable. In standard neural MT training, the reference is: (1) used in the training objective; and (2) con- ditioned on as the previous target token.1 We ap- proximate the full space of possible translations by: (1) training the MT …

Content truncated to 500 words