Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2536 - 2541 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation Sho Takase and Tatsuya Hiraoka and Naoaki Okazaki Tokyo Institute of Technology {sho.takase@nlp., tatsuya.hiraoka@nlp., okazaki@}c.titech.ac.jp Abstract Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation mod- els. In previous subword regularizations, we use multiple segmentations in the training pro- cess but use only one segmentation in the in- ference. In this study, we propose an inference strategy to address this discrepancy. The pro- posed strategy approximates the marginalized likelihood by using multiple segmentations in- cluding the most plausible segmentation and several sampled segmentations. Because the proposed strategy aggregates predictions from several segmentations, we can regard it as a sin- gle model ensemble that does not require any additional cost for training. Experimental re- sults show that the proposed strategy improves the performance of models trained with sub- word regularization in low-resource machine translation tasks. 1 Introduction Subword regularizations are the technique to make a model robust to segmentation errors by using multiple subword segmentations instead of only the most plausible segmentation during the train- ing process (Kudo, 2018; Provilkov et al., 2020). Previous studies demonstrated that subword reg- ularizations improve the performance of LSTM- based encoder-decoders and Transformers in vari- ous machine translation datasets, especially in low- resource settings (Kudo, 2018). However, previous subword regularizations con- tain the discrepancy between the training and in- ference. In the training process, we stochasti- cally re-segment a given sequence into subwords based on statistics such as the uni-gram language model (Kudo, 2018). Thus, we use multiple seg- mentations for each input sequence. In contrast, we use only the most plausible segmentation in the inference phase. We expect that we can improve the performance by solving this discrepancy. To solve this discrepancy, we propose an infer- ence strategy that uses multiple subword segmen- tations. We construct multiple subword segmen- tations for an input in the same manner as that in the training process, and then aggregate the pre- dictions from each segmentation. Therefore, our proposed inference strategy can be regarded as a single model ensemble using multiple segmenta- tions. Figure 1 illustrates the overview of previous methods and our proposed inference strategy. We conduct experiments on several machine translation datasets. Experimental results show that the proposed strategy improves the performance of a subword regularized model without any addi- tional costs in the training procedure when the sub- word regularization significantly contributes to the performance, i.e., in low-resource settings. More- over, we indicate that our strategy can be combined with a widely used model ensemble technique. 2 Subword Regularization Our proposed strategy is based on a model trained with subword regularization. Thus, we briefly de- scribe subword regularization in this section. Kudo (2018) proposed subword regularization to improve the robustness of a neural machine transla- tion model. Let X and Y be the source and target …

Content truncated to 500 words