Charagram: Embedding Words and Sentences via Character n-grams

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1504–1515, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics CHARAGRAM: Embedding Words and Sentences via Character n-grams John Wieting Mohit Bansal Kevin Gimpel Karen Livescu Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {jwieting,mbansal,kgimpel,klivescu}@ttic.edu Abstract We present CHARAGRAM embeddings, a sim- ple approach for learning character-based compositional models to embed textual se- quences. A word or sentence is represented using a character n-gram count vector, fol- lowed by a single nonlinear transformation to yield a low-dimensional embedding. We use three tasks for evaluation: word simi- larity, sentence similarity, and part-of-speech tagging. We demonstrate that CHARAGRAM embeddings outperform more complex archi- tectures based on character-level recurrent and convolutional neural networks, achieving new state-of-the-art performance on several simi- larity tasks.1 1 Introduction Representing textual sequences such as words and sentences is a fundamental component of natural language understanding systems. Many functional architectures have been proposed to model compo- sitionality in word sequences, ranging from sim- ple averaging (Mitchell and Lapata, 2010; Iyyer et al., 2015) to functions with rich recursive struc- ture (Socher et al., 2011; Zhu et al., 2015; Tai et al., 2015; Bowman et al., 2016). Most work uses words as the smallest units in the compositional ar- chitecture, often using pretrained word embeddings or learning them specifically for the task of inter- est (Tai et al., 2015; He et al., 2015). Some prior work has found benefit from using character-based compositional models that encode 1Trained models and code are available at http://ttic. uchicago.edu/˜wieting. arbitrary character sequences into vectors. Exam- ples include recurrent neural networks (RNNs) and convolutional neural networks (CNNs) on character sequences, showing improvements for several NLP tasks (Ling et al., 2015a; Kim et al., 2015; Balles- teros et al., 2015; dos Santos and Guimar˜aes, 2015). By sharing subword information across words, char- acter models have the potential to better represent rare words and morphological variants. Our approach, CHARAGRAM, uses a much sim- pler functional architecture. We represent a charac- ter sequence by a vector containing counts of char- acter n-grams, inspired by Huang et al. (2013). This vector is embedded into a low-dimensional space using a single nonlinear transformation. This can be interpreted as learning embeddings of character n-grams, which are learned so as to produce effec- tive sequence embeddings when a summation is per- formed over the character n-grams in the sequence. We consider three evaluations: word similar- ity, sentence similarity, and part-of-speech tagging. On multiple word similarity datasets, CHARAGRAM outperforms RNNs and CNNs, achieving state-of- the-art performance on SimLex-999 (Hill et al., 2015). When evaluated on a large suite of sentence- level semantic textual similarity tasks, CHARA- GRAM embeddings again outperform the RNN and CNN architectures as well as the PARAGRAM- PHRASE embeddings of Wieting et al. (2016). We also consider English part-of-speech (POS) tagging using the bidirectional long short-term memory tag- ger of Ling et al. (2015a). The three architectures reach similar performance, though CHARAGRAM converges fastest to …

Content truncated to 500 words