Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1716–1731 June 6–11, 2021. ©2021 Association for Computational Linguistics 1716 Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models Yuxuan Lai1,2,∗, Yijia Liu3, Yansong Feng1,2,† , Songfang Huang3 and Dongyan Zhao1,2 1Wangxuan Institute of Computer Technology, Peking University, China 2The MOE Key Laboratory of Computational Linguistics, Peking University, China 3Alibaba Group {erutan, fengyansong, zhaody}@pku.edu.cn {yanshan.lyj, songfang.hsf}@alibaba-inc.com Abstract Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese — Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sen- tence in a multi-granularity manner. Specif- ically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We de- sign a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked seg- ment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice struc- tures, and the improvement comes from the ex- ploration of redundant information and multi- granularity representations.1 1 Introduction Pre-trained Language Models (PLMs) have achieved promising results in many Chinese Nat- ural Language Understanding (NLU) tasks (Cui et al., 2019; Liu et al., 2020; Sun et al., 2020). These models take a sequence of fine-grained units — Chinese characters — as the input, following the English PLMs’ practice (Devlin et al., 2019, BERT). ∗Work done during an internship at Alibaba DAMO Academy. † Corresponding author. 1Our code will be available at https://github. com/alibaba/pretrained-language-models/ LatticeBERT. � study � investigate � alive � live � very � fill � solid ��� graduate student �� research �� life �� fulfilling Figure 1: An illustration of the word lattice for sentence 研究生活很充实/Research Life is very fulfilling. However, the meanings of many Chinese words cannot be fully understood through direct compo- sitions of their characters’ meanings. For exam- ple, 老板/boss does not mean 老/elder 板/board.2 The importance of word-level inputs in Chinese has been addressed in different tasks, including relation classification (Li et al., 2019), short text matching (Lai et al., 2019; Chen et al., 2020; Lyu et al., 2021), trigger detection (Lin et al., 2018), and named entity recognition (Zhang and Yang, 2018; Gui et al., 2019; Li et al., 2020a). The coarse- grained inputs benefit these tasks by introducing word-level semantics with multi-granularity repre- sentations, which is potentially complementary in character-level Chinese PLMs. In this work, we discuss how to pre-train a Chi- nese PLM over a word lattice structure to exploit multi-granularity inputs. We argue …

Content truncated to 500 words