Learning Syntactic Categories Using Paradigmatic Representations of Word Context

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 940–951, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Learning Syntactic Categories Using Paradigmatic Representations of Word Context Mehmet Ali Yatbaz Enis Sert Artificial Intelligence LaboratoryDeniz Yuret Koc¸ University, ˙Istanbul, Turkey {myatbaz,esert,dyuret}@ku.edu.tr Abstract We investigate paradigmatic representations of word context in the domain of unsupervised syntactic category acquisition. Paradigmatic representations of word context are based on potential substitutes of a word in contrast to syntagmatic representations based on prop- erties of neighboring words. We compare a bigram based baseline model with several paradigmatic models and demonstrate signif- icant gains in accuracy. Our best model based on Euclidean co-occurrence embedding com- bines the paradigmatic context representation with morphological and orthographic features and achieves 80% many-to-one accuracy on a 45-tag 1M word corpus. 1 Introduction Grammar rules apply not to individual words (e.g. dog, eat) but to syntactic categories of words (e.g. noun, verb). Thus constructing syntactic categories (also known as lexical or part-of-speech categories) is one of the fundamental problems in language ac- quisition. Syntactic categories represent groups of words that can be substituted for one another without alter- ing the grammaticality of a sentence. Linguists iden- tify syntactic categories based on semantic, syntac- tic, and morphological properties of words. There is also evidence that children use prosodic and phono- logical features to bootstrap syntactic category ac- quisition (Ambridge and Lieven, 2011). However there is as yet no satisfactory computational model that can match human performance. Thus identify- ing the best set of features and best learning algo- rithms for syntactic category acquisition is still an open problem. Relationships between linguistic units can be classified into two types: syntagmatic (concerning positioning), and paradigmatic (concerning substitu- tion). Syntagmatic relations determine which units can combine to create larger groups and paradig- matic relations determine which units can be sub- stituted for one another. Figure 1 illustrates the paradigmatic vs syntagmatic axes for words in a simple sentence and their possible substitutes. In this study, we represent the paradigmatic axis directly by building substitute vectors for each word position in the text. The dimensions of a substi- tute vector represent words in the vocabulary, and the magnitudes represent the probability of occur- rence in the given position. Note that the substitute vector for a word position (e.g. the second word in Fig. 1) is a function of the context only (i.e. “the cried”), and does not depend on the word that does actually appear there (i.e. “man”). Thus substi- Figure 1: Syntagmatic vs. paradigmatic axes for words in a simple sentence (Chandler, 2007). 940 tute vectors represent individual word contexts, not word types. We refer to the use of features based on substitute vectors as paradigmatic representations of word context. Our preliminary experiments indicated that using context information alone without the identity or the features of the target word (e.g. using dimension- ality reduction and clustering on substitute vectors) has limited success …

Content truncated to 500 words