Multidirectional Associative Optimization of Function-Specific Word Representations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2872–2882 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2872 Multidirectional Associative Optimization of Function-Specific Word Representations Daniela Gerz♠♦ Ivan Vuli´c♠ Marek Rei♣ Roi Reichart♥ Anna Korhonen♠ ♠Language Technology Lab, University of Cambridge ♦PolyAI Limited, London ♣Department of Computing, Imperial College London ♥Faculty of Industrial Engineering and Management, Technion, IIT {dan,ivan}@poly-ai.com, marek.rei@imperial.ac.uk roiri@ie.technion.ac.il, alk23@cam.ac.uk Abstract We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb- Object (SVO) structures. Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO com- positions lie close together. The model re- tains information about word group member- ship even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure. We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional prefer- ence and event similarity. The results indi- cate that the combinations of representations learned with our task-independent model out- perform task-specific architectures from prior work, while reducing the number of parame- ters by up to 95%. 1 Introduction Word representations are in ubiquitous usage across all areas of natural language processing (NLP) (Col- lobert et al., 2011; Chen and Manning, 2014; Mela- mud et al., 2016). Standard approaches rely on the distributional hypothesis (Harris, 1954; Sch¨utze, 1993) and learn a single word vector space based on word co-occurrences in large text corpora (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017). This purely context-based training produces general word representations that capture the broad notion of semantic relatedness and con- flate a variety of possible semantic relations into a single space (Hill et al., 2015; Schwartz et al., 2015). However, this mono-faceted view of mean- ing is a well-known deficiency in NLP applications (Faruqui, 2016; Mrkˇsi´c et al., 2017) as it fails to dis- tinguish between fine-grained word associations. In this work we propose to learn a joint function- specific word vector space that accounts for the study eat need food help support assistance subject art science researcher chicken scientist implementation cat chicken Figure 1: Illustration of three neighbourhoods in a function-specific space trained for the SVO structure (marked �(S), �(V), $(O)). The space is optimised such that vectors for plausible SVO compositions will be close. Note that one word can have several vectors, for example chicken can occur both as S and O. different roles and functions a word can take in text. The space can be trained for a specific structure, such as SVO, and each word in a particular role will have a separate representation. Vectors for plausi- ble SVO compositions will then be optimized to lie close together, as illustrated by Figure 1. For exam- ple, the verb vector study will be close to plausible subject vectors researcher or scientist and object vectors subject or art. For words that …

Content truncated to 500 words