Coding Textual Inputs Boosts the Accuracy of Neural Networks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1350–1360, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1350 Coding Textual Inputs Boosts the Accuracy of Neural Networks Abdul Rafae Khan⋆, Jia Xu⋆ ∗, and Weiwei Sun† ⋆Department of Computer Science, Stevens Institute of Technology {akhan4, jxu70}@stevens.edu †Department of Computer Science and Technology, University of Cambridge† ws390@cam.ac.uk Abstract Natural Language Processing (NLP) tasks are usually performed word by word on textual inputs. We can use arbitrary symbols to rep- resent the linguistic meaning of a word and use these symbols as inputs. As “alterna- tives” to a text representation, we introduce Soundex, MetaPhone, NYSIIS, logogram to NLP, and develop fixed-output-length coding and its extension using Huffman coding. Each of those codings combines different charac- ter/digital sequences and constructs a new vo- cabulary based on codewords. We find that the integration of those codewords with text pro- vides more reliable inputs to Neural-Network- based NLP systems through redundancy than text-alone inputs. Experiments demonstrate that our approach outperforms the state-of- the-art models on the application of machine translation, language modeling, and part-of- speech tagging. The source code is available at https://github.com/abdulrafae/coding nmt. 1 Introduction We introduce novel coding schemes on the inputs of Neural-Network-based Natural Language Pro- cessing (NN-NLP) that significantly boost the accu- racy in three applications. The inputs of NN-NLP rely on observable forms of mental representations of linguistic expressions, and allow alternative de- signs. For example, both logographic kanji and syllabic kana represent Japanese words, and emoti- cons and emojis can express sentiments. These showcase that alternative human language repre- sentation than text is possible and highlight a com- mon belief of most linguists: the relationship be- tween the mental representations and their phono- logical forms is highly arbitrary, even though a ∗Jia Xu is the corresponding author of this paper. †This work was completed when Weiwei Sun was at Peking University. non-arbitrary (de Saussure, 1916) mapping exists for some special cases, e.g., the bouba/kiki effect. In our work, we ask – Are there alternative forms of mental representation in addition to text as we see in Japanese and Internet language to help lan- guage understanding in NN-NLP? To answer this question, we blend concepts from linguistic phonetics, grammatology, and the statis- tics of Zipf law to find alternative language repre- sentations to text. More precisely, we code a textual word either naturally or artificially by exploring dif- ferent facets of human languages, from phonetic and logogram codings to new coding constructions generalizable to all languages. Natural codings in- spire the finding of artificial codings, which in turn helps us understand and explain natural codings. All of our codings reinforce NLP inputs by re- constructing the character/symbol sequence of a word in various ways with a new alphabet. These variants and their “decomposition” are expressive because they contain insightful information about linguistic patterns in units smaller than words and even smaller than characters. For example, in the logogram Wubi (that lists in a coded form …

Content truncated to 500 words