Picking the Amateur’s Mind - Predicting Chess Player Strength from Game Annotations

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 311–321, Dublin, Ireland, August 23-29 2014. Picking the Amateur’s Mind – Predicting Chess Player Strength from Game Annotations Christian Scheible Institute for Natural Language Processing University of Stuttgart, Germany scheibcn@ims.uni-stuttgart.de Hinrich Sch¨utze Center for Information and Language Processing University of Munich, Germany Abstract Results from psychology show a connection between a speaker’s expertise in a task and the lan- guage he uses to talk about it. In this paper, we present an empirical study on using linguistic evidence to predict the expertise of a speaker in a task: playing chess. Instructional chess litera- ture claims that the mindsets of amateur and expert players differ fundamentally (Silman, 1999); psychological science has empirically arrived at similar results (e.g., Pfau and Murphy (1988)). We conduct experiments on automatically predicting chess player skill based on their natural lan- guage game commentary. We make use of annotated chess games, in which players provide their own interpretation of game in prose. Based on a dataset collected from an online chess forum, we predict player strength through SVM classification and ranking. We show that using textual and chess-specific features achieves both high classification accuracy and significant correlation. Finally, we compare our findings to claims from the chess literature and results from psychology. 1 Introduction It has been recognized that the language used when describing a certain topic or activity may differ strongly depending on the speaker’s level of expertise. As shown in empirical experiments in psychology (e.g., Solomon (1990), Pfau and Murphy (1988)), a speaker’s linguistic choices are influenced by the way he thinks about the topic. While writer expertise has been addressed previously, we know of no work that uses linguistic indicators to rank experts. We present a study on predicting chess expertise from written commentary. Chess is a particularly interesting task for predicting expertise: First, using data from competitive online chess, we can compare and rank players within a well-defined ranking system. Second, we can collect textual data for experi- mental evaluation from web resources, eliminating the need for manual annotation. Third, there is a large amount of terminology associated with chess, which we can exploit for n-gram based classification. Chess is difficult for humans because it requires long-term foresight (strategy) as well as the capacity for internally simulating complicated move sequences (calculation and tactics). For these reasons, the game for a long time remained challenging even for computers. Players have thus developed general principles of chess strategy on which many expert players agree. The dominant expert view is that the understanding of fundamental strategical notions, supplemented by the ability of calculation, is the most important skill of a chess player. A good player develops a long-term plan for the course of the game. This view is the foundation of many introductory works to chess (e.g., Capablanca (1921), one of the earliest works). Silman (1999) presents games he played with chess students, analyzing their commentary about the progress of the game. …

Content truncated to 500 words