Grouping Words with Semantic Diversity

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3217–3228 June 6–11, 2021. ©2021 Association for Computational Linguistics 3217 Grouping Words with Semantic Diversity Karine Chubarian§ = Abdul Rafae Khan†= Anastasios Sidiropoulos§ = Jia Xu† ∗ †Department of Computer Science, Stevens Institute of Technology {akhan4, jxu70}@stevens.edu §Department of Computer Science and Technology, University of Illinois at Chicago {kchuba2, sidiropo}@uic.edu Abstract Deep Learning-based NLP systems can be sensitive to unseen tokens and hard to learn with high-dimensional inputs, which critically hinder learning generalization. We introduce an approach by grouping input words based on their semantic diversity to simplify input language representation with low ambiguity. Since the semantically diverse words reside in different contexts, we are able to substi- tute words with their groups and still distin- guish word meanings relying on their con- texts. We design several algorithms that com- pute diverse groupings based on random sam- pling, geometric distances, and entropy max- imization, and we prove formal guarantees for the entropy-based algorithms. Experimen- tal results show that our methods generalize NLP models and demonstrate enhanced accu- racy on POS tagging and LM tasks and sig- nificant improvements on medium-scale ma- chine translation tasks, up to +6.5 BLEU points. Our source code is available at https://github.com/abdulrafae/dg. 1 Introduction Natural Language Understanding has seen remark- able success with the rise of Deep Learning. How- ever, human languages’ variety and richness result in high-dimensional inputs to NLP models, increas- ing learning complexity and error rates. First, open- vocabulary inputs inevitably bring rare and out- of-Vocabulary words (OOVs). Second, network complexity increases with input dimension, specifi- cally the “curse of dimensionality” makes learning difficult on medium and small datasets. This paper addresses these limitations by intro- ducing new grouping methods to compute alterna- tive language representations that simplify textual inputs. We currently have alternative language rep- resentations, such as Pinyin, Metaphone, logogram, ∗Author ordered alphabetically. Jia Xu is the correspond- ing author of this paper. and Emoji, that exist in natural languages and that have been shown to improve various NLP applica- tions (Du and Way, 2017; Liu et al., 2018; Khan et al., 2020). While these representations can help, they are not developed for NLP performance. Our goal is to design algorithms for computing new lan- guage representations specifically to enhance NLP performance in this work. We ask: “Can we compute a generalized language repre- sentation to improve NLP applications?” An intuitive approach to answering this ques- tion is to group similar words in training and test sets and replace each word with its group. A word grouping viewed as a many-to-one mapping func- tion can significantly reduce the vocabulary size that lowers the input feature dimensions leading to a generalized NLP model learning. For example, let us take two sentences: (a) “you ask me.”; (b) “she tells me.” There are five words “ask”, “she”, “tells”, “me”, and “you” in the vo- cabulary. Grouping words into “A” and “B” will reduce the …

Content truncated to 500 words