Variable Mini-Batch Sizing and Pre-Trained Embeddings

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 680–686 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Variable Mini-Batch Sizing and Pre-Trained Embeddings Mostafa Abdou and Vladan Glonˇc´ak and Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics mostafahany56@gmail.com, {gloncak,bojar}@ufal.mff.cuni.cz Abstract This paper describes our submission to the WMT 2017 Neural MT Training Task. We modified the provided NMT system in order to allow for interrupting and con- tinuing the training of models. This al- lowed mid-training batch size decremen- tation and incrementation at variable rates. In addition to the models with variable batch size, we tried different setups with pre-trained word2vec embeddings. Aside from batch size incrementation, all our ex- periments performed below the baseline. 1 Introduction We participated in the WMT 2017 NMT Training Task, experimenting with pre-trained word em- beddings and mini-batch sizing. The underly- ing NMT system (Neural Monkey, Helcl and Li- bovick´y, 2017) was provided by the task organiz- ers (Bojar et al., 2017), including the training data for English to Czech translation. The goal of the task was to find training criteria and training data layout which leads to the best translation quality. The provided NMT system is based on an atten- tional encoder-decoder (Bahdanau, Cho, and Ben- gio, 2014) and utilizes BPE for vocabulary size re- duction to allow handling open vocabulary (Sen- nrich, Haddow, and Birch, 2016). We modified the provided NMT system in order to allow for interruption and continuation of the training process by saving and reloading variable files. This did not result in any noticeable change in the learning. Furthermore, it allowed for mid- training mini-batch size decrementation and incre- mentation at variable rates. As our main experiment, we tried to employ pre-trained word embeddings to initialize embed- dings in the model on the source side (monolin- gually trained embeddings) and on both source and target sides (bilingually trained embeddings). Section 1.1 describes our baseline system. Sec- tion 2 examines the pre-trained embeddings and Section 3 the effect of batch size modifications. Further work and conclusion (Sections 4 and 5) close the paper. 1.1 The Baseline System Our baseline model was trained using the provided NMT system and the provided data, including the given word splits of BPE (Sennrich, Haddow, and Birch, 2016). Of the two available configurations, we selected the 4GB one for most experiments to fit the limits of GPU cards available at MetaCen- trum.1 This configuration uses a maximum sen- tence length of 50, word embeddings of size 300, hidden layers of size 350, and clips the gradient norm to 1.0. We used a mini-batch size of 60 for this model. Due to resource limitations at MetaCentrum, the training had to be interrupted after a week of training. We modified Neural Monkey to en- able training continuation by saving and loading the model and we always submitted the continued training as a new job. When tested with restarts ev- ery few hours, …

Content truncated to 500 words