Predictability of Distributional Semantics in Derivational Word Formation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1285–1296, Osaka, Japan, December 11-17 2016. Predictability of Distributional Semantics in Derivational Word Formation Sebastian Pad´o∗ Aur´elie Herbelot+ Max Kisselew∗ Jan ˇSnajder† ∗ Institute for Natural Language Processing, University of Stuttgart {sebastian.pado,max.kisselew}@ims.uni-stuttgart.de + Center for Mind/Brain Sciences, University of Trento {aurelie.herbelot}@unitn.it † Faculty of Electrical Engineering and Computing, University of Zagreb jan.snajder@fer.hr Abstract Compositional distributional semantic models (CDSMs) have successfully been applied to the task of predicting the meaning of a range of linguistic constructions. Their performance on semi- compositional word formation process of (morphological) derivation, however, has been extremely variable, with no large-scale empirical investigation to date. This paper fills that gap, performing an analysis of CDSM predictions on a large dataset (over 30,000 German derivationally related word pairs). We use linear regression models to analyze CDSM performance and obtain insights into the linguistic factors that influence how predictable the distributional context of a derived word is going to be. We identify various such factors, notably part of speech, argument structure, and semantic regularity. 1 Introduction Compositional models of distributional semantics, or CDSMs (Mitchell and Lapata, 2010; Erk and Pad´o, 2008; Baroni et al., 2014; Coecke et al., 2010), have established themselves as a standard tool in computational semantics. Building on traditional distributional semantic models for individual words (Turney and Pantel, 2010), they are generally applied to compositionally compute phrase meaning by defining combination operations on the meaning of the phrase’s constituents. CDSMs have also been co-opted by the deep learning community for tasks including sentiment analysis (Socher et al., 2013) and machine translation (Hermann and Blunsom, 2014). A more recent development is the use of CDSMs to model meaning-related phenomena above and below syntactic structure; here, the term “composition” is used more generally to apply to processes of meaning combination from multiple linguistic units, e.g., above and below syntactic structure. Above the sentence level, such models attempt to predict the unfolding of discourse (Kiros et al., 2015). Below the word level, CDSMs have been applied to model word formation processes like compounding (church + tower → church tower) and (morphological) derivation (Lazaridou et al., 2013) (favor + able → favorable). More concretely, given a distributional representation of a basis and a derivation pattern (typically an affix), the task of the CDSM is to predict a distributional representation of the derived word, without being provided with any additional information. Interest in the use of CDSMs in this context comes from the observation that derived words are often less frequent than their bases (Hay, 2003), and in the extreme case even completely novel; consequently, distributional evidence is often unreliable and sometimes unavailable. This is confirmed by Luong et al. (2013) who compare the performance of different types of word embeddings on a word similarity task and achieve poorer performance on data sets containing rarer and more complex words. Due to the Zipfian distribution there are many more rare than frequent word types in a corpus, which …

Content truncated to 500 words