Contextualized Word Representations for Multi-Sense Embedding

Contextualized Word Representations for Multi-Sense Embedding Kazuki Ashihara Osaka University ashihara.kazuki@ist.osaka-u.ac.jp Yuki Arase Osaka University arase@ist.osaka-u.ac.jp Tomoyuki Kajiwara Osaka University kajiwara@ids.osaka-u.ac.jp Satoru Uchida Kyushu University uchida@flc.kyushu-u.ac.jp Abstract Distributed word representations are used in many natural language processing tasks. When dealing with ambiguous words, it is de- sired to generate multi-sense embeddings, i.e., multiple representations per word. Therefore, several methods have been proposed to gen- erate different word representations based on parts of speech or topic, but these methods tend to be too coarse to deal with ambiguity. In this paper, we propose methods to generate multiple word representations for each word based on dependency structure relations. In order to deal with the data sparseness prob- lem due to the increase in the size of vocabu- lary, the initial value for each word represen- tations is determined using pre-trained word representations. It is expected that the repre- sentations of low frequency words will remain in the vicinity of the initial value, which will in turn reduce the negative effects of data sparse- ness. Extensive evaluation results confirm the effectiveness of our methods that signif- icantly outperformed state-of-the-art methods for multi-sense embeddings. Detailed analysis of our method shows that the data sparseness problem is resolved due to the pre-training. 1 Introduction Distributed word representations by neural net- works (Mikolov et al., 2013a; Mikolov et al., 2013b; Pennington et al., 2014; Levy and Gold- berg, 2014; Bojanowski et al., 2017), which are one of the implementations of the distributional hy- pothesis (Harris, 1954), have shown surprising ef- fectiveness in representing semantic or syntactic in- formation of words by dense vectors. They have become the standard language resource for many natural language processing tasks, such as machine translation (Sutskever et al., 2014), text classifica- tion (Mikolov and Com, 2014) and lexical substitu- tion (Melamud et al., 2015). Widely used methods for word embedding including CBOW (Continuous Bag-of-Words) (Mikolov et al., 2013b) and SGNS (Skip-gram with Negative Sampling) (Mikolov et al., 2013a) generate a single word representation per word. Hence several meanings of the word are mixed in the representation, which significantly af- fect the downstream usability. To solve this problem, several methods (Nee- lakantan et al., 2014; Paetzold and Specia, 2016; Fadaee et al., 2017; Athiwaratkun and Wilson, 2017) to generate multi-sense embeddings, i.e., represen- tations for each word sense are proposed. Because word sense disambiguation itself is still an open problem, these studies use approximate approaches. Paetzold and Specia (2016) use parts of speech to distinguish functionally different word senses, while Fadaee et al. (2017) propose to use topics. How- ever, either parts of speech or topics are too coarse to distinguish word senses. In the following exam- ples, both sentences have the topic of food, and the parts of speech of the words soft are both adjec- tive.1 ex.1) I ate a soft cheese. ex.2) I drank soft drinks. Although the previous studies regard the senses of 1We use the typewriter font to indicate examples. PACLIC 32 28 32nd Pacific Asia Conference …

Content truncated to 500 words