Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 56–68 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1006 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 56–68 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1006 Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules Ivan Vuli´c1 , Nikola Mrkši´c1, Roi Reichart2 Diarmuid Ó Séaghdha3, Steve Young1 , Anna Korhonen1 1 University of Cambridge 2 Technion, Israel Institute of Technology 3 Apple Inc. {iv250,nm480,sjy11,alk23}@cam.ac.uk doseaghdha@apple.com roiri@ie.technion.ac.il Abstract Morphologically rich languages accentu- ate two properties of distributional vec- tor space models: 1) the difficulty of in- ducing accurate representations for low- frequency word forms; and 2) insensitivity to distinct lexical relations that have simi- lar distributional signatures. These effects are detrimental for language understanding systems, which may infer that inexpensive is a rephrasing for expensive or may not as- sociate acquire with acquires. In this work, we propose a novel morph-fitting procedure which moves past the use of curated seman- tic lexicons for improving distributional vector spaces. Instead, our method injects morphological constraints generated using simple language-specific rules, pulling in- flectional forms of the same word close to- gether and pushing derivational antonyms far apart. In intrinsic evaluation over four languages, we show that our approach: 1) improves low-frequency word estimates; and 2) boosts the semantic quality of the entire word vector collection. Finally, we show that morph-fitted vectors yield large gains in the downstream task of dialogue state tracking, highlighting the importance of morphology for tackling long-tail phe- nomena in language understanding tasks. 1 Introduction Word representation learning has become a re- search area of central importance in natural lan- guage processing (NLP), with its usefulness demon- strated across many application areas such as pars- ing (Chen and Manning, 2014; Johannsen et al., 2015), machine translation (Zou et al., 2013), and many others (Turian et al., 2010; Collobert et al., 2011). Most prominent word representation tech- niques are grounded in the distributional hypothe- sis (Harris, 1954), relying on word co-occurrence information in large textual corpora (Curran, 2004; Turney and Pantel, 2010; Mikolov et al., 2013; Mnih and Kavukcuoglu, 2013; Levy and Goldberg, 2014; Schwartz et al., 2015, i.a.). Morphologically rich languages, in which “sub- stantial grammatical information. . . is expressed at word level” (Tsarfaty et al., 2010), pose specific challenges for NLP. This is not always considered when techniques are evaluated on languages such as English or Chinese, which do not have rich mor- phology. In the case of distributional vector space models, morphological complexity brings two chal- lenges to the fore: 1. Estimating Rare Words: A single lemma can have many different surface realisations. Naively treating each realisation as a separate word leads to sparsity problems and a failure to exploit their shared semantics. On the other hand, lemma- tising the entire corpus can obfuscate the differ- ences that exist between different word forms even though they share …

Content truncated to 500 words