Evaluating a Joint Training Approach for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora on Lower-resource Languages

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 12–25 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 12 Can Transformer Language Models Predict Psychometric Properties? Antonio Laverghetta Jr., Animesh Nighojkar, Jamshidbek Mirzakhalov, and John Licato Advancing Machine and Human Reasoning (AMHR) Lab Department of Computer Science and Engineering University of South Florida Tampa, FL, USA {alaverghett,anighojkar,mirzakhalov,licato}@usf.edu Abstract Transformer-based language models (LMs) continue to advance state-of-the-art perfor- mance on NLP benchmark tasks, including tasks designed to mimic human-inspired “com- monsense” competencies. To better under- stand the degree to which LMs can be said to have certain linguistic reasoning skills, re- searchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numer- ous human participants and LMs (transformer- and non-transformer-based) on a broad diag- nostic test of linguistic competencies. We then use the responses to calculate standard psy- chometric properties of the items in the di- agnostic test, using the human responses and the LM responses separately. We then deter- mine how well these two sets of predictions match. We find cases in which transformer- based LMs predict psychometric properties consistently well in certain categories but con- sistently poorly in others, thus providing new insights into fundamental similarities and dif- ferences between human and LM reasoning.1 1 Introduction The current generation of transformer-based lan- guage models (TLMs) (Vaswani et al., 2017) con- tinues to surpass expectations, consistently achiev- ing state-of-the-art results on many natural lan- guage processing (NLP) benchmark tasks. Espe- cially surprising is their remarkable performance on benchmark tasks designed to assess “common- sense” reasoning (e.g., Wang et al., 2018, 2019), 1Code and data to reproduce our experiments can be found on Github: https://github.com/Advancing-Machine-Human- Reasoning-Lab/transformer-psychometrics possibly owing to their ability to encode and re- trieve a surprising amount of structural knowledge (Goldberg, 2019; Hu et al., 2020; Cui et al., 2020; Petroni et al., 2019; Davison et al., 2019), despite initial worries that all connectionist language mod- els in general would suffer the same limitations as previous generations (Sun, 1992, 1995; McClel- land, 1995; Klahr, 1999; McLaughlin, 2009). Understanding how TLMs reason is a complex task made more difficult by the fact that the sizes of contemporary TLMs are so large as to effectively render them black boxes. As such, researchers are continually searching for new methods to under- stand the strengths and limitations of TLMs. One promising approach is to draw from psychometrics, a sub-field of psychology particularly suited to deal- ing with perhaps the most mysterious black box of them all: the human mind. Psychometrics is con- cerned with psychological measurement—i.e., how to measure latent attributes like reasoning skills, attitudes, and personality traits. Psychometricians have developed tools to measure such properties even when the mechanisms that give rise …

Content truncated to 500 words