A Simple and Effective Usage of Word Clusters for CBOW Model

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 80–86 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 80 A Simple and Effective Usage of Word Clusters for CBOW Model Yukun Feng1, Chenlong Hu1, Hidetaka Kamigaito1, Hiroya Takamura1,2 and Manabu Okumura1 1Tokyo Institute of Technology 2National Institute of Advanced Industrial Science and Technology (AIST) {yukun,huchenlong,kamigaito,oku}@lr.pi.titech.ac.jp takamura@pi.titech.ac.jp Abstract We propose a simple and effective method for incorporating word clusters into the Continu- ous Bag-of-Words (CBOW) model. Specifi- cally, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of fre- quent words and a small amount of cluster em- beddings, which will be fine-tuned in down- stream tasks. We empirically show our replac- ing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings. 1 Introduction Word embeddings have been widely applied to various natural language processing (NLP) tasks. These embeddings can be pretrained on a large cor- pus and carry useful semantic information. One of the most well-known methods for obtaining word embeddings is based on Continuous Bag-of-Words (CBOW) (Mikolov et al., 2013a) and there have been many research efforts to extend it. In this paper, we focus on incorporating word clusters into CBOW model. Each word cluster consists of words that function similarly. By aggre- gating such words, we can alleviate data sparsity, even though each of those words is infrequent. In the past few years, word clusters have been applied to various tasks, such as named-entity recognition (Ritter et al., 2011), machine translation (Wuebker et al., 2013) and parsing (Kong et al., 2014). Many word clustering algorithms can be applied to a raw corpus with different languages and help us obtain word clusters easily without additional language resources. In our method, we keep only very frequent words and replace the other words with their clusters for both input and output words in the CBOW model. This is motivated by the fact that word clusters are more reliable than infrequent words. Thus, only very frequent word embeddings and a small amount of cluster embeddings are produced as the output. When fine-tuning the trained embeddings on downstream tasks, the embeddings of infrequent words within one cluster are initialized by the em- bedding of their cluster to increase the coverage of pretrained word embeddings. Since word embeddings are usually trained on the large-scale dataset. For making clusters on the large-scale dataset, we choose bidirectional, inter- polated, refining, and alternating (BIRA) predictive exchange algorithm (Dehdari et al., 2016)1 as our clustering method. Because BIRA was reported to be faster than many other methods. Notably, it can produce 800 clusters on 1 billion English tokens in 1.4 hours. We evaluate our cluster-incorporated word em- beddings2 on downstream tasks, in which fine- tuning of word embeddings …

Content truncated to 500 words