Watching a Language Model Learning Chess

Proceedings of Recent Advances in Natural Language Processing, pages 1369–1379 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_153 1369 Watching a Language Model Learning Chess Andreas St¨ockl University of Applied Sciences Upper Austria Digital Media Department Hagenberg, Austria andreas.stoeckl@fh-hagenberg.at Abstract We analyse how a transformer-based language model learns the rules of chess from text data of recorded games. We show how it is possible to investigate how the model capacity and the available number of training data influence the learning success of a language model with the help of chess-specific metrics. With these met- rics, we show that more games used for train- ing in the studied range offers significantly bet- ter results for the same training time. How- ever, model size does not show such a clear influence. It is also interesting to observe that the usual evaluation metrics for language mod- els, predictive accuracy and perplexity, give no indication of this here. Further examina- tion of trained models reveals how they store information about board state in the activa- tions of neuron groups, and how the over- all sequence of previous moves influences the newly-generated moves. 1 Introduction Language models are now used for a variety of ap- plications that are not, or not directly, related to Nat- ural Language Processing tasks, and process data that is not text Parmar et al. (2018); Huang et al. (2018); Dhariwal et al. (2020). Lu et al. (2021) used a so-called Frozen Pretrained Transformer (FPT) to study finetuning on a variety of sequence clas- sification tasks spanning numerical computation, vision, and protein fold prediction. In this article, we use state-of-the-art methods for language models in an area that at first glance does not seem to be an application area for them, namely the area of computer chess. This is because clear rules determine what happens here, rather than the ambiguities and vagueness that charac- terise language. Brown et al. (2020) demonstrated, among other things, that a language model with increasing model capacity is able to learn the rules of arith- metic to a certain degree by training it with the data of crawled websites. In the process, elemen- tary operations were learned in a certain number space, but not beyond. Is this limitation due to the lack of capacity of the model, insufficient training time or training data that did not contain sufficient information? In Nogueira et al. (2021), it was demonstrated that regardless of the number of parameters and training examples, Transformer Vaswani et al. (2017) models are unable to learn addition rules that are independent of the length of the numbers seen during training. To test the ability of language models to learn rules, and assess the influence of model size, train- ing time and available training data, we use the commonly-stressed field of “computer chess” as an example. We investigate whether a language model is able to learn the rules of chess only from the records of games played by humans. The test area is well suited for studying the train- ing …

Content truncated to 500 words