Few-shot learning through contextual data augmentation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1049–1062 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1049 Few-shot learning through contextual data augmentation Farid Arthaud ∗ Ecole Normale Sup´erieure farid.arthaud@ens.psl.eu Rachel Bawden ∗ Inria rachel.bawden@inria.fr Alexandra Birch University of Edinburgh A.Birch@ed.ac.uk Abstract Machine translation (MT) models used in in- dustries with constantly changing topics, such as translation or news agencies, need to adapt to new data to maintain their performance over time. Our aim is to teach a pre-trained MT model to translate previously unseen words ac- curately, based on very few examples. We propose (i) an experimental setup allowing us to simulate novel vocabulary appearing in human-submitted translations, and (ii) corre- sponding evaluation metrics to compare our approaches. We extend a data augmentation approach using a pre-trained language model to create training examples with similar con- texts for novel words. We compare different fine-tuning and data augmentation approaches and show that adaptation on the scale of one to five examples is possible. Combining data aug- mentation with randomly selected training sen- tences leads to the highest BLEU score and ac- curacy improvements. Impressively, with only 1 to 5 examples, our model reports better ac- curacy scores than a reference system trained with on average 313 parallel examples. 1 Introduction News agencies dealing with articles in multiple languages often rely on machine translation (MT) to provide an initial translation, which a journal- ist reworks into a final article. This can involve changing the structure or phrasing, but also cor- recting (or post-editing) mistranslated words or ex- pressions, which can frequently occur when new topics emerge, bringing new vocabulary that has been rarely or never seen in the data used for train- ing. The willingness of a journalist to use MT technology is dependent on the general quality of the models, but also on whether they can learn from the journalist’s corrections, to avoid them having to correct the same errors time and time again. ∗Work done at University of Edinburgh Various strategies have been explored to learn from a journalist’s post-edits. One option is to use an automatic post-editing (APE) model trained on the journalist’s post-edits. However, state-of-the- art APE systems (Junczys-Dowmunt and Grund- kiewicz, 2017) typically require large numbers of post-edits for training, which are rarely available or hard to generate (particularly for low-resource languages). An alternative, commonly used strat- egy is to fine-tune models to in-domain data, but this is prone to overfitting (Miceli Barone et al., 2017). More advanced ways of continually learn- ing through fine-tuning have been explored, select- ing similar training instances based on their similar- ity with test sentences (Li et al., 2018; Turchi et al., 2017). These methods achieve good results accord- ing to automatic MT metrics but can also overfit when training is continued. So far there has been little focus on the speed of adaptation required—so as to minimize the number of human interventions required—and the trade-off between successfully adapting …

Content truncated to 500 words