Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit?

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5708–5713 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5708 Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? Emmanuele Chersoni1, Ludovica Pannitto2, Enrico Santus3, Alessandro Lenci4, Chu-Ren Huang1 The Hong Kong Polytechnic University 1, University of Trento2, Massachussetts Institute of Technology3, University of Pisa4 Chinese and Bilingual Studies, 11 Yuk Choi Road, Hung Hom, Hong Kong (China)1 Centre for Mind/Brain Sciences, Corso Bettini 31, Rovereto (Italy)2 MIT CSAIL, 32 Vassar Street, Cambridge, MA (United States)3 University of Pisa, Via Santa Maria 36, 56126 Pisa (Italy)4 {emmanuele.chersoni,churen.huang}@polyu.edu.hk1, ludovica.pannitto@unitn.it2, esantus@mit.edu3, alessandro.lenci@unipi.it4 Abstract While neural embeddings represent a popular choice for word representation in a wide variety of NLP tasks, their usage for thematic fit modeling has been limited, as they have been reported to lag behind syntax-based count models. In this paper, we propose a complete evaluation of count models and word embeddings on thematic fit estimation, by taking into account a larger number of parameters and verb roles and introducing also dependency-based embeddings in the comparison. Our results show a complex scenario, where a determinant factor for the performance seems to be the availability to the model of reliable syntactic information for building the distributional representations of the roles. Keywords: Semantics, Cognitive Methods, Statistical and Machine Learning Methods 1. Introduction In recent years, vectors derived from neural network train- ing have quickly replaced the old, count-based Distribu- tional Semantic Models (DSMs) as a de facto standard for word representation in NLP.1 Tools such as Word2Vec (Mikolov et al., 2013a; Mikolov et al., 2013b) have pro- vided the research community with an efficient and scal- able method for training vector representations, generally referred to as word embeddings. Moreover, the embeddings have been reported to have an advantage over the old count models also in terms of performance in several NLP tasks (Baroni et al., 2014).2 In this scenario, thematic fit estimation represents an excep- tion. Concretely, the task consists in estimating a typicality score for a filler noun given a verb role (e.g., a system has to predict how plausible a cake is as a patient of the verb to eat). It is generally evaluated by assessing the correla- tion between collections of human judgements and DSM outputs, and it represents an important benchmark for the capacity of the models of capturing compositional meaning (Lenci, 2018). In a systematic comparison between count-based models and neural embeddings, (Baroni et al., 2014) showed that the latter outperform the former in almost all the evaluation tasks, while in the thematic fit task they were vastly out- performed by dependency-based count models. The results by Baroni and colleagues were later confirmed by (Say- 1Throughout the entire paper, we will refer to the traditional distributional vectors based on co-occurrence counts as count models (Baroni et al., 2014). 2See however also the research works by (Levy et al., 2015; Lebret and Collobert, 2015; Gamallo, 2017), …

Content truncated to 500 words