Embedding Syntax and Semantics of Prepositions via Tensor Decomposition

Proceedings of NAACL-HLT 2018, pages 896–906 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Embedding Syntax and Semantics of Prepositions via Tensor Decomposition Hongyu Gong, Suma Bhat, Pramod Viswanath {hgong6, spbhat2, pramodv}@illinois.edu Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign, USA Abstract Prepositions are among the most frequent words in English and play complex roles in the syntax and semantics of sentences. Not surprisingly, they pose well-known difficulties in automatic processing of sentences (prepo- sitional attachment ambiguities and idiosyn- cratic uses in phrases). Existing methods on preposition representation treat preposi- tions no different from content words (e.g., word2vec and GloVe). In addition, recent studies aiming at solving prepositional attach- ment and preposition selection problems de- pend heavily on external linguistic resources and use dataset-specific word representations. In this paper we use word-triple counts (one of the triples being a preposition) to capture a preposition’s interaction with its attachment and complement. We then derive preposi- tion embeddings via tensor decomposition on a large unlabeled corpus. We reveal a new ge- ometry involving Hadamard products and em- pirically demonstrate its utility in paraphras- ing phrasal verbs. Furthermore, our preposi- tion embeddings are used as simple features in two challenging downstream tasks: prepo- sition selection and prepositional attachment disambiguation. We achieve results compara- ble to or better than the state-of-the-art on mul- tiple standardized datasets. 1 Introduction Prepositions are a linguistically closed class com- prising some of the most frequent words; they play an important role in the English language since they encode rich syntactic and semantic informa- tion. Many preposition-related tasks are challeng- ing in computational linguistics because of their polysemous nature and flexible usage patterns. An accurate understanding and representation of prepositions’ linguistic role is key to several im- portant NLP tasks such as grammatical error cor- rection and prepositional phrase attachment. A first-order approach is to represent prepositions as real-valued vectors via word embeddings such as word2vec (Mikolov et al., 2013) and GloVe (Pen- nington et al., 2014). Word embeddings have brought a renaissance in NLP research; they have been very successful in capturing word similarities as well as analo- gies (both syntactic and semantic) and are now mainstream in nearly all downstream NLP tasks (such as question-answering (Chen et al., 2017)). Despite this success, available literature does not highlight any specific properties of word embed- dings of prepositions. Indeed, many of the com- mon prepositions have very similar vector rep- resentations as shown in Table 1 for preposition vectors trained using word2vec and GloVe (Ten- sor embedding is our proposed representation for prepositions). While this suggests that using avail- able representations for prepositions diminishes the distinguishing aspect between prepositions, one could hypothesize that this is primarily be- cause standard word embedding algorithms treat prepositions no different from other content words such as verbs and nouns, i.e., embeddings are cre- ated based on co-occurrences with other words. However, prepositions are very frequent and co- occur with nearly all words, which means that …

Content truncated to 500 words