Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1520–1530, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation Wang Ling Tiago Lu´ıs Lu´ıs Marujo Ram´on Fernandez Astudillo Silvio Amir Chris Dyer Alan W Black Isabel Trancoso L2F Spoken Systems Lab, INESC-ID, Lisbon, Portugal Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA Instituto Superior T´ecnico, Lisbon, Portugal {lingwang,lmarujo,cdyer,awb}@cs.cmu.edu {ramon.astudillo,samir,tmcl,isabel.trancoso}@inesc-id.pt Abstract We introduce a model for construct- ing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word rep- resentation models that have independent vectors for each word type, our model requires only a single vector per char- acter type and a fixed set of parame- ters for the compositional model. De- spite the compactness of this model and, more importantly, the arbitrary nature of the form–function relationship in lan- guage, our “composed” word representa- tions yield state-of-the-art results in lan- guage modeling and part-of-speech tag- ging. Benefits over traditional baselines are particularly pronounced in morpholog- ically rich languages (e.g., Turkish). 1 Introduction Good representations of words are important for good generalization in natural language process- ing applications. Of central importance are vec- tor space models that capture functional (i.e., se- mantic and syntactic) similarity in terms of ge- ometric locality. However, when word vectors are learned—a practice that is becoming increas- ingly common—most models assume that each word type has its own vector representation that can vary independently of other model compo- nents. This paper argues that this independence assumption is inherently problematic, in particular in morphologically rich languages (e.g., Turkish). In such languages, a more reasonable assumption would be that orthographic (formal) similarity is evidence for functional similarity. However, it is manifestly clear that similarity in form is neither a necessary nor sufficient condi- tion for similarity in function: small orthographic differences may correspond to large semantic or syntactic differences (butter vs. batter), and large orthographic differences may obscure nearly per- fect functional correspondence (rich vs. affluent). Thus, any orthographically aware model must be able to capture non-compositional effects in addi- tion to more regular effects due to, e.g., morpho- logical processes. To model the complex form– function relationship, we turn to long short-term memories (LSTMs), which are designed to be able to capture complex non-linear and non-local dy- namics in sequences (Hochreiter and Schmidhu- ber, 1997). We use bidirectional LSTMs to “read” the character sequences that constitute each word and combine them into a vector representation of the word. This model assumes that each charac- ter type is associated with a vector, and the LSTM parameters encode both idiosyncratic lexical and regular morphological knowledge. To evaluate our model, we use a vector- based model for part-of-speech (POS) tagging and for language modeling, and we report ex- periments on these tasks in several languages comparing to baselines that use more tradi- tional, orthographically-unaware parameteriza- tions. These experiments show: (i) our character- based model is able to generate …

Content truncated to 500 words