Data and Parameter Scaling Laws for Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5915 Data and Parameter Scaling Laws for Neural Machine Translation Mitchell Gordon Johns Hopkins University mitchg@jh.edu Kevin Duh Johns Hopkins University kevinduh@cs.jhu.edu Jared Kaplan Johns Hopkins University jaredk@jhu.edu Abstract We observe that the development cross- entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these re- sults, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs. 1 Introduction As training neural networks becomes an organi- zational and multi-million dollar venture (Brown et al., 2020), it is imperative to quantifiably pre- dict the benefits of scaling up neural networks. In this paradigm, machine learning is an engineering effort, in which money can buy resources (data, compute) and the main concern is to predict return- on-investment (ROI) while avoiding bottlenecks. Recent work has observed that the cross entropy loss of neural language models and other autore- gressive generative models scales like a power law in the amount of training data, compute, and num- ber of model parameters over several orders of mag- nitude (Hestness et al., 2019; Kaplan et al., 2020; Henighan et al., 2020). Similar intuitions exist in the realm of supervised MT: doubling the amount of parallel training data leads to roughly a fixed improvement in BLEU in both phrase-based statis- tical MT (Irvine and Callison-Burch, 2013; Turchi et al., 2008) and neural MT (Koehn and Knowles, 2017; Sennrich and Zhang, 2019). In Section 2, we show that these MT intuitions can be quantified and explained via cross-entropy power law scaling; using a handful of experiments on small subsets of MT datasets, we precisely pre- dict the performance of large systems trained on orders of magnitude more data. In Section 3, we demonstrate how these trends might be utilized to make ROI predictions when annotating more data for low-resource language pairs. 2 Machine Translation Scaling Laws To investigate the predictability of MT system per- formance as parameters/data increase, we train many Transformers of various sizes (Table 2) on randomly selected subsets of data ( 1 2, 1 4, 1 8, ...) for several standard MT datasets. The smallest data subsets contain ∼0.1% of the total data available. We use three language pairs in our experiments: Figure 1: The development cross-entropy vs. the amount of data used to train each model from Section 2. Each point is colored by the number of non-embedding parameters in the model. The best fit of Equation 1 via least- squares regression for each language pair is shown as the dotted line. 5916 Lang Pair # Train Examples αN log NC αD log DC Ru-En 51.1 M 0.11 21.62 0.38 13.81 De-En 28.3 M 0.13 18.81 0.35 13.43 Zh-En 35.9 M 0.12 19.90 0.43 12.73 Table 1: The …

Content truncated to 500 words