A Closer Look at Parameter Contributions When Training Neural Language and Translation Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 4788–4800 October 12–17, 2022. 4788 A Closer Look at Parameter Contributions When Training Neural Language and Translation Models Raúl Vázquez♠ Hande Celikkanat♠ Vinit Ravishankar♢ Mathias Creutz♠ Jörg Tiedemann♠ ♠Department of Digital Humanities, University of Helsinki ♢Language Technology Group, Department of Informatics, University of Oslo ♠{firstname.lastname@}@helsinki.fi ♢vinitr@ifi.uio.no Abstract We analyze the learning dynamics of neural language and translation models using Loss Change Allocation (LCA), an indicator that enables a fine-grained analysis of parameter updates when optimizing for the loss function. In other words, we can observe the contribu- tions of different network components at train- ing time. In this article, we systematically study masked language modeling, causal lan- guage modeling, and machine translation. We show that the choice of training objective leads to distinctive optimization procedures, even when performed on comparable Transformer architectures. We demonstrate how the vari- ous Transformer parameters are used during training, supporting that the feed-forward com- ponents of each layer are the main contributors to the optimization procedure. Finally, we find that the learning dynamics are not affected by data size and distribution but rather determined by the learning objective. 1 Introduction Neural models and Transformers in particular have achieved a great performance in almost every nat- ural language processing (NLP) task. However, they largely remain black-box systems despite var- ious efforts to analyze them. Much work has been devoted to understanding the dense represen- tations that are created during training (Pavlick, 2022; Saphra, 2021) in an attempt to see whether known linguistic properties are present and how they are expressed in the model (Vuli´c et al., 2020; Raganato et al., 2020; Serrano and Smith, 2019). However, the intrinsic dynamics of the training procedure itself have not been analyzed in depth for highly complex network architectures. In this paper, we take a closer look into the process of parameter optimization, aiming at • identifying the network components that con- tribute to the optimization of the loss function, • revealing the effects of learning objectives and network components on training dynamics, • detecting the impact of training data size and distribution on learning dynamics and param- eter updates. In order to enable a systematic comparison, we set out with a standard architecture that we train from scratch for each individual task. We use the Transformer architecture (Vaswani et al., 2017) with its common parametrization and size, and train it with three popular learning objectives: masked language modeling (MLM) (Taylor, 1953; Devlin et al., 2019), causal language modeling (CLM), and neural machine translation (NMT). The latter is an interesting special case as it is designed as a condi- tional language model in a sequence-to-sequence architecture with encoder and decoder components. For the analysis, we use Loss Change Allocation (LCA) (Lan et al., 2019), an indicator of the con- tribution of individual parameters to the decrease of the total loss. Its accumulative properties pro- vide us with a tool to perform a fine-grained anal- ysis of the opaque optimization process …

Content truncated to 500 words