Assessing the Limits of the Distributional Hypothesis in Semantic Spaces: Trait-based Relational Knowledge and the Impact of Co-occurrences

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 173 - 185 July 14-15, 2022 ©2022 Association for Computational Linguistics Assessing the Limits of the Distributional Hypothesis in Semantic Spaces: Trait-based Relational Knowledge and the Impact of Co-occurrences Mark Anderson PIN Caerdydd Prifysgol Caerdydd AndersonM8@caerdydd.ac.uk Jose Camacho-Collados Cardiff NLP Cardiff University CamachoColladosJ@cardiff.ac.uk Abstract The increase in performance in NLP due to the prevalence of distributional models and deep learning has brought with it a reciprocal de- crease in interpretability. This has spurred a focus on what neural networks learn about natu- ral language with less of a focus on how. Some work has focused on the data used to develop data-driven models, but typically this line of work aims to highlight issues with the data, e.g. highlighting and offsetting harmful biases. This work contributes to the relatively untrod- den path of what is required in data for mod- els to capture meaningful representations of natural language. This entails evaluating how well English and Spanish semantic spaces cap- ture a particular type of relational knowledge, namely the traits associated with concepts (e.g. bananas-yellow), and exploring the role of co- occurrences in this context. 1 Introduction Vector space models have been the main driving force behind progress in NLP. Most work in this area, either in the form of static or contextualised embeddings, has been based on co-occurrence statistics and largely driven by the distributional hy- pothesis (Harris, 1954; Firth, 1957). This has also resulted in these representations seemingly cap- turing certain relational knowledge, such as word analogies (Mikolov et al., 2013b; Gittens et al., 2017). In this context, Chiang et al. (2020) found that the ability of word embeddings to evaluate analogies was not greatly impaired by removing co-occurrences related to relational pairs. This sug- gests there are limits to how the distributional hy- pothesis impacts the encoding of relational knowl- edge. We extend this line of work by focusing on the relational knowledge of concepts and traits. We also creep beyond English by translating concept and traits used in one of our datasets into Spanish. Contributions: (1) We show that there is no im- pact on the ability of semantic spaces to predict whether a pair of embeddings corresponds to a trait-concept pair or to predict what traits a given concept has when removing co-occurrences of con- cepts and traits. (2) We developed a freely available dataset that can be used for further trait-based re- lational knowledge analyses for English and Span- ish.1 2 Related work What models learn Evaluation of neural semantic spaces has focused on what knowledge they cap- ture with a slew of work showing that some knowl- edge of analogies can be seen by applying simple transformations (Mikolov et al., 2013b; Levy and Goldberg, 2014; Arora et al., 2016; Paperno and Baroni, 2016; Gittens et al., 2017; Ethayarajh et al., 2019). Others have investigated what syntactic in- formation neural semantic spaces seem to capture with most showing that they do capture something deeper …

Content truncated to 500 words