Learning Character-level Compositionality with Visual Features

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2059–2068 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1188 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2059–2068 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1188 Learning Character-level Compositionality with Visual Features Frederick Liu1, Han Lu1, Chieh Lo2, Graham Neubig1 1Language Technology Institute 2Electrical and Computer Engineering Carnegie Mellon University, Pittsburgh, PA 15213 {fliu1,hlu2,gneubig}@cs.cmu.edu chiehl@andrew.cmu.edu Abstract Previous work has modeled the composi- tionality of words by creating character- level models of meaning, reducing prob- lems of sparsity for rare words. However, in many writing systems compositionality has an effect even on the character-level: the meaning of a character is derived by the sum of its parts. In this paper, we model this effect by creating embeddings for characters based on their visual charac- teristics, creating an image for the charac- ter and running it through a convolutional neural network to produce a visual char- acter embedding. Experiments on a text classification task demonstrate that such model allows for better processing of in- stances with rare characters in languages such as Chinese, Japanese, and Korean. Additionally, qualitative analyses demon- strate that our proposed model learns to focus on the parts of characters that carry semantic content, resulting in embeddings that are coherent in visual space. 1 Introduction Compositionality—the fact that the meaning of a complex expression is determined by its structure and the meanings of its constituents—is a hall- mark of every natural language (Frege and Austin, 1980; Szab´o, 2010). Recently, neural models have provided a powerful tool for learning how to com- pose words together into a meaning representation of whole sentences for many downstream tasks. This is done using models of various levels of sophistication, from simpler bag-of-words (Iyyer et al., 2015) and linear recurrent neural network (RNN) models (Sutskever et al., 2014; Kiros et al., 2015), to more sophisticated models using tree- �� ���� ���� �� �� Kalb� Kälber� a� Do� Do'(polite)� Calf� Calves� �� �� �� �� �� �� �� �� Laurel� Whale� Salmon� Salmon� gui� jing� gui� gui� (a)� (b)� (c)� (d)� han'da� ham''ni'''da� Figure 1: Examples of character-level composi- tionality in (a, b) Chinese, (c) Korean, and (d) Ger- man. The red part of the characters are shared, and affects the pronunciation (top) or meaning (bot- tom). structured (Socher et al., 2013) or convolutional networks (Kalchbrenner et al., 2014). In fact, a growing body of evidence shows that it is essential to look below the word-level and con- sider compositionality within words themselves. For example, several works have proposed mod- els that represent words by composing together the characters into a representation of the word it- self (Ling et al., 2015; Zhang et al., 2015; Dhingra et al., 2016). Additionally, for languages with pro- ductive word formation (such as agglutination and compounding), models calculating morphology- sensitive word representations have been found ef- fective (Luong et …

Content truncated to 500 words