Analyzing Word Embedding Through Structural Equation Modeling

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1823–1832 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1823 Analyzing Word Embedding Through Structural Equation Modeling Namgi Han12, Katsuhiko Hayashi3, Yusuke Miyao3 1The Graduate University for Advanced Studies, SOKENDAI, Japan 2National Institute of Informatics, Japan 3The University of Tokyo, Japan namgi@nii.ac.jp, {katsuhiko-h, yusuke}@is.s.u-tokyo.ac.jp Abstract Many researchers have tried to predict the accuracies of extrinsic evaluation by using intrinsic evaluation to evaluate word embedding. The relationship between intrinsic and extrinsic evaluation, however, has only been studied with simple correlation analysis, which has difficulty capturing complex cause-effect relationships and integrating external factors such as the hyperparameters of word embedding. To tackle this problem, we employ partial least squares path modeling (PLS-PM), a method of structural equation modeling developed for causal analysis. We propose a causal diagram consisting of the evaluation results on the BATS, VecEval, and SentEval datasets, with a causal hypothesis that linguistic knowledge encoded in word embedding contributes to solving downstream tasks. Our PLS-PM models are estimated with 600 word embeddings, and we prove the existence of causal relations between linguistic knowledge evaluated on BATS and the accuracies of downstream tasks evaluated on VecEval and SentEval in our PLS-PM models. Moreover, we show that the PLS-PM models are useful for analyzing the effect of hyperparameters, including the training algorithm, corpus, dimension, and context window, and for validating the effectiveness of intrinsic evaluation. Keywords: Word Embedding, Intrinsic Evaluation, Extrinsic Evaluation, Structural Equation Modeling, Partial Least Squares Path Modeling 1. Introduction Word embedding is an indispensable tool for a variety of natural language processing (NLP) tasks, yet it is still un- clear why and how it contributes to achieving high accuracy on NLP tasks. A series of extrinsic experiments has proven the effectiveness of word embedding on downstream NLP tasks such as syntactic analysis and semantic textual simi- larity (Nayak et al., 2016; Conneau and Kiela, 2018). On the other hand, intrinsic evaluation of word embedding, as in word similarity (Bruni et al., 2014; Hill et al., 2015) and word analogy tasks (Mikolov et al., 2013; Gladkova et al., 2016), has been proposed for assessing what linguistic knowledge it encodes. Previous studies (Chiu et al., 2016; Rogers et al., 2018; Wang et al., 2019) tried to prove this intuition by using correlation analysis. Correlation analysis does not, however, assume any causal hypothesis, there- fore it is hard to extract any cause-effect relationships from those studies. Hence, we investigate causal relations between the accura- cies of intrinsic and extrinsic evaluation, by applying par- tial least squares path modeling (PLS-PM) (Wold, 1982), a method of structural equation modeling. PLS-PM is a widely accepted method in social science disciplines for analyzing causal relations among observed and latent vari- ables (Henseler et al., 2014). In PLS-PM, hypothetical causal relations are given together via a causal diagram, and the strengths of the causal relations are then estimated by fitting the causal diagram to observed data. This method enables …

Content truncated to 500 words