UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 201–211 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 201 UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database Canwen Xu1∗, Tao Ge2, Chenliang Li3, Furu Wei2 1 University of California, San Diego 2 Microsoft Research Asia 3 Wuhan University 1 cxu@ucsd.edu 2 {tage,fuwei}@microsoft.com 3 cllee@whu.edu.cn Abstract Chinese and Japanese share many charac- ters with similar surface morphology. To better utilize the shared knowledge across the languages, we propose UnihanLM, a self-supervised Chinese-Japanese pretrained masked language model (MLM) with a novel two-stage coarse-to-fine training approach. We exploit Unihan, a ready-made database constructed by linguistic experts to first merge morphologically similar characters into clus- ters. The resulting clusters are used to re- place the original characters in sentences for the coarse-grained pretraining of the MLM. Then, we restore the clusters back to the original characters in sentences for the fine- grained pretraining to learn the representation of the specific characters. We conduct ex- tensive experiments on a variety of Chinese and Japanese NLP benchmarks, showing that our proposed UnihanLM is effective on both mono- and cross-lingual Chinese and Japanese tasks, shedding light on a new path to exploit the homology of languages.1 1 Introduction Recently, Pretrained Language Models have shown promising performance on many NLP tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019c; Lan et al., 2020). Many attempts have been made to train a model that supports mul- tiple languages. Among them, Multilingual BERT (mBERT) (Devlin et al., 2019) is released as a part of BERT. It directly adopts the same model ar- chitecture and training objective, and is trained on Wikipedia in different languages. XLM (Lam- ple and Conneau, 2019) is proposed with an ad- ditional language embedding and a new training ∗ This work was done during Canwen’s internship at Microsoft Research Asia. 1The code and pretrained weights are available at https: //github.com/JetRunner/unihan-lm. JA 台1風2は熱3帯4低気5圧6の一種7です。 T-ZH 颱1風2是熱3帶4氣5旋的一種7。 S-ZH 台1风2是热3带4低气5压6的一种7。 EN Typhoon is a type of tropical depression. Table 1: A sentence example in Japanese (JA), Tradi- tional Chinese (T-ZH) and Simplified Chinese (S-ZH) with its English translation (EN). The characters that already share the same Unicode are marked with an underline. In this work, we further match characters with identical meanings but different Unicode, then merge them. Characters eligible to be merged together are marked with the same superscript. objective (translation language modeling, TLM). XLM-R (Conneau et al., 2019) has a larger size and is trained with more data. Based on XLM, Uni- coder (Huang et al., 2019) collects more data and uses multi-task learning to train on three supervised tasks. The census of cross-lingual approaches is to al- low lexical information to be shared between lan- guages. XLM and mBERT exploit shared lexical information by Byte Pair Encoding (BPE) (Sen- nrich et al., 2016) and WordPiece (Wu et al., …

Content truncated to 500 words