Nonsymbolic Text Representation

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 785–796, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Nonsymbolic Text Representation Hinrich Sch¨utze CIS, LMU Munich, Germany inquiries@cislmu.org Abstract We introduce the first generic text repre- sentation model that is completely non- symbolic, i.e., it does not require the avail- ability of a segmentation or tokenization method that attempts to identify words or other symbolic units in text. This applies to training the representations as well as to using them in an application. We demon- strate better performance than prior work on entity typing and text denoising. 1 Introduction Character-level models can be grouped into three classes. (i) End-to-end models learn a separate model on the raw character (or byte) input for each task; these models estimate task-specific pa- rameters, but no representation of text that would be usable across tasks is computed. Throughout this paper, we refer to r(x) as the “representa- tion” of x only if r(x) is a generic rendering of x that can be used in a general way, e.g., across tasks and domains. The activation pattern of a hidden layer for a given input sentence in a mul- tilayer perceptron (MLP) is not a representation according to this definition if it is not used out- side of the MLP. (ii) Character-level models of words derive a representation of a word w from the character string of w, but they are symbolic in that they need text segmented into tokens as input. (iii) Bag-of-character-ngram models, bag- of-ngram models for short, use character ngrams to encode sequence-of-character information, but sequence-of-ngram information is lost in the rep- resentations they produce. Our premise is that text representations are needed in NLP. A large body of work on word embeddings demonstrates that a generic text rep- resentation, trained in an unsupervised fashion on large corpora, is useful. Thus, we take the view that group (i) models, end-to-end learning with- out any representation learning, is not a good gen- eral approach for NLP. We distinguish training and utilization of the text representation model. We use “training” to refer to the method by which the model is learned and “utilization” to refer to the application of the model to a piece of text to compute a representa- tion of the text. In many text representation mod- els, utilization is trivial. For example, for word embedding models, utilization amounts to a sim- ple lookup of a word to get its precomputed em- bedding. However, for the models we consider, utilization is not trivial and we will discuss differ- ent approaches. Both training and utilization can be either sym- bolic or nonsymbolic. We define a symbolic ap- proach as one that is based on tokenization, i.e., a segmentation of the text into tokens. Symbol iden- tifiers (i.e., tokens) can have internal structure – a tokenizer may recognize tokens like “to and fro” and “London-based” that contain delimiters – and may be …

Content truncated to 500 words