Improving Neural Language Processing with Named Entities

Proceedings of Recent Advances in Natural Language Processing, pages 940–949 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_107 940 Improving Neural Language Processing with Named Entities Kyoumoto Matsushita Fujitsu Ltd. / Tokyo, Japan m.kyoumoto@fujitsu.com Takuya Makino Fujitsu Ltd. / Tokyo, Japan makino.takuya@fujitsu.com Tomoya Iwakura Fujitsu Ltd. / Tokyo, Japan iwakura.tomoya@fujitsu.com Abstract Pretraining-based neural network models have demonstrated state-of-the-art (SOTA) perfor- mances on natural language processing (NLP) tasks. The most frequently used sentence rep- resentation for neural-based NLP methods is a sequence of subwords that is different from the sentence representation of non-neural methods that are created using basic NLP technologies, such as part-of-speech (POS) tagging, named entity (NE) recognition, and parsing. Most neural-based NLP models receive only vec- tors encoded from a sequence of subwords obtained from an input text. However, ba- sic NLP information, such as POS tags, NEs, parsing results, etc, cannot be obtained explic- itly from only the large unlabeled text used in pretraining-based models. This paper ex- plores use of NEs on two Japanese tasks; docu- ment classification and headline generation us- ing Transformer-based models, to reveal the ef- fectiveness of basic NLP information. The ex- perimental results with eight basic NEs and ap- proximately 200 extended NEs show that NEs improve accuracy although a large pretraining- based model trained using 70 GB text data was used. 1 Introduction In statistical NLP technologies, which were widely employed before neural-based NLP emerged, ba- sic information, such as POS tags and NEs were often used as features for document classifica- tion (Higashinaka et al., 2012) and other NLP tasks. However, since neural-based NLP technologies have emerged with state-of-the-art (SOTA) perfor- mances, basic NLP technologies, such as POS tag- ging and NE recognition are no longer used for obtaining features from given text. This is because most neural-based NLP methods attain higher ac- curacy using only sentence representation encoded by a pretraining model learned from the large scale unlabeled text. However, we think there are some rooms that basic information, such as POS tags and NEs con- tribute to recent neural-based NLP even if large scale pretrained models such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) are used for obtaining sentence representations. One of the reasons is that POS tags and NEs are usually obtained from outputs of analyzers trained from labeled training data created by human-beings for each specific purpose. For example, NE rec- ognizers can identify single words or phrases with their classes, such as PERSON, ORGANIZATION, and so on, which are not explicitly given from pre- trained models. Therefore, we think different kinds of information compared with pretraining ones are obtained from outputs of such NLP tools and we expect such information contributes to further im- prove accuracy. We propose the incorporation of basic NLP in- formation to neural NLP architectures and evalu- ate their effectiveness on two Japanese NLP tasks. In this paper, two NE categories, eight basic NEs (Sekine and Isahara, 2000) and approximately 200 extended NEs (Sekine and Nobata, 2004), are con- …

Content truncated to 500 words