Generation and Evaluation of Concept Embeddings Via Fine-Tuning Using Automatically Tagged Corpus

Generation and Evaluation of Concept Embeddings Via Fine-Tuning Using Automatically Tagged Corpus Kanako Komiya Daiki Yaginuma Masayuki Asahara Hiroyuki Shinnou Ibaraki University 4-12-1, Nakanarusawa, Hitachi, Ibaraki, 316-8511, Japan {kanako.komiya.nlp, 18nm740n, hiroyuki.shinnou.0828}@vc.ibaraki.ac.jp National Institute for Japanese Language and Linguistics 10-2 Midoricho, Tachikawa, Tokyo, Japan masayu-a@ninjal.ac.jp Abstract Word embeddings are used in various fields of natural language processing. The use of word embeddings and concept or word sense em- beddings demonstrated effectiveness in many tasks, such as machine translation and text summarization. However, it is difficult to ob- tain a sufficiently large concept-tagged cor- pus, as the annotation of concept-tags is time- consuming. Therefore, in this paper, we pro- pose a method for generating concept em- beddings of Word List by Semantic Princi- ples, a Japanese thesaurus, using both a cor- pus tagged by an all-words word sense dis- ambiguation (WSD) system and a manually tagged corpus. We generated concept em- beddings via fine-tuning using both an auto- matically tagged corpus and a small manually tagged corpus. In this paper, we propose a novel method of evaluating concept embed- dings using the tree structure of Word List by Semantic Principles. Experiments revealed the effectiveness of fine-tuning. The best per- formance was achieved when the concept em- beddings were initially trained with a corpus tagged by an all-words WSD system and re- trained with a manually tagged corpus. 1 Introduction In this paper, we propose a technique for generat- ing concept embeddings using fine-tuning and two types of corpora. In recent years, word embeddings, which are distributed representations of words with low-dimensional vectors, and concept 1 (or word 1Concept refers to a meaning unit of Word List by Semantic Principles. sense) embeddings demonstrated their effectiveness in a number of tasks, such as machine translation and text summarization. Word embeddings are usually generated using text corpora. It is possible to generate concept em- beddings by the same method used to generate word embeddings if the word sequence (i.e., text cor- pus) is replaced with a concept sequence constructed from a concept-tagged corpus. However, it is dif- ficult to obtain a sufficiently large concept-tagged corpus because the annotation of concept tags is time-consuming. There have been several studies that assigned word senses using the all-words word sense disambiguation (WSD) method (Edmonds and Cotton, 2001), (Snyder and Palmer, 2004), (Nav- igli et al., 2007), (Iacobacci et al., 2016), (Raganato et al., 2017a), (Raganato et al., 2017b), (Suzuki et al., 2018), (Shinnou et al., 2018). As a result, it is possible to create a concept-tagged corpus using the methods proposed in these studies. However, the re- sults of all-words WSD systems are not always cor- rect; therefore, an automatically tagged corpus cre- ated via all-words WSD may not be suitable for gen- erating concept embeddings. In this paper, we generate concept embeddings of Word List by Semantic Principles (WLSP) (Na- tional Institute for Japanese Language and Linguis- tics, 1964), a Japanese thesaurus, from manually and automatically tagged corpora. First, concept embed- dings are generated from a concept-tagged …

Content truncated to 500 words