Misspelling Oblivious Word Embeddings

Proceedings of NAACL-HLT 2019, pages 3226–3234 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3226 Misspelling Oblivious Word Embeddings Bora Edizel Facebook AI, London edizel@fb.edu Aleksandra Piktus Facebook AI, London piktus@fb.com Piotr Bojanowski Facebook AI, Paris bojanowski@fb.com Rui Ferreira∗ Spotify, London ruif@spotify.com Edouard Grave Facebook AI, Paris egrave@fb.com Fabrizio Silvestri Facebook AI, London fsilvestri@fb.com Abstract In this paper we present a method learn word embeddings that are resilient toto misspellings. Existing word embeddings have limited applicability to malformed texts, which contain a non-negligible amount of out- of-vocabulary words. We propose a method combining FastText with subwords and a su- pervised task of learning misspelling patterns. In our method, misspellings of each word are embedded close to their correct variants. We train these embeddings on a new dataset we are releasing publicly. Finally, we experimentally show the advantages of this approach on both intrinsic and extrinsic NLP tasks using public test sets. 1 Introduction Word embeddings constitute a building block of many practical applications across NLP and re- lated disciplines. Techniques such as Word2Vec (Mikolov et al., 2013a,b) and GloVe (Pennington et al., 2014) have been extensively used in prac- tice. One of their drawbacks, however, is that they cannot provide embeddings for words that have not been observed at training time, i.e. Out-Of- Vocabulary (OOV) words. In real-world tasks, the input text is often generated by people and misspellings, a common source of OOV words, are frequent (e.g. (Cucerzan and Brill, 2004) re- port that misspellings appear in up to 15% of web search queries). As a consequence, the quality of downstream applications of word embeddings in real-world scenarios diminishes. Simply allowing the inclusion of misspellings into corpora and vocabularies in existing method- ologies might not provide satisfactory results. The sparsity of misspellings would most likely prevent ∗ This work was carried out when the author was work- ing as an employee at Facebook London. their embeddings from demonstrating any inter- esting properties. Trying to balance the represen- tation of misspellings with the representation of correctly spelled variants in training data by arti- ficially introducing misspelled variants for every word in the corpus would on the other hand cause up to an exponential growth in the size of the train- ing data, making training of the models infeasible. To address this deficiency, we propose Mis- spelling Oblivious (word) Embeddings (MOE), a new model combining FastText (Bojanowski et al., 2017) with a supervised task which embeds mis- spellings close to their correct variants. We carry out experiments on well established tasks and on their variants adapted to the misspellings prob- lem. We also propose new methods of evaluating embeddings specifically designed to capture their quality on misspelled words. We train MOE em- beddings on a new dataset we are releasing pub- licly. Finally, we experimentally show the advan- tages of this approach on both intrinsic and extrin- sic NLP tasks using public test sets. Summarizing, we propose the following contributions: • a novel problem and a …

Content truncated to 500 words