A Robustly Optimized BERT Pre-training Approach with Post-training
3590Authors
nan
Abstract
AbstractIn the paper we present a ‘pre-training’+‘post-training’+‘fine-tuning’ three-stage paradigm which is a supplementary framework for the standard ‘pre-training’+‘fine-tuning’ languagemodel approach. Furthermore based on three-stage paradigm we present a language modelnamed PPBERT. Compared with original BERT architecture that is based on the standard two-stage paradigm we do not fine-tune pre-trained model directly but rather post-train it on the domain or task related dataset first which helps to better incorporate task-awareness knowl-edge and domain-awareness knowledge within pre-trained model also from the training datasetreduce bias. Extensive experimental results indicate that proposed model improves the perfor-mance of the baselines on 24 NLP tasks which includes eight GLUE benchmarks eight Su-perGLUE benchmarks six extractive question answering benchmarks. More remarkably our proposed model is a more flexible and pluggable model where post-training approach is able to be plugged into other PLMs that are based on BERT. Extensive ablations further validate the effectiveness and its state-of-the-art (SOTA) performance. The open source code pre-trained models and post-trained models are available publicly.
Keywords
Full Content
中 中 中文 文 文词 词 词语 语 语离 离 离合 合 合现 现 现象 象 象识 识 识别 别 别研 研 研究 究 究 周 周 周露 露 露1 ， ， ，曲 曲 曲维 维 维光 光 光1,2,∗ ， ， ，魏 魏 魏庭 庭 庭新 新 新2,3 ， ， ，周 周 周俊 俊 俊生 生 生1 ， ， ，李 李 李斌 斌 斌2 ， ， ，顾 顾 顾彦 彦 彦慧 慧 慧1 (1.南京师范大学 计算机与电子信息学院/人工智能学院，江苏省 南京市 210023； 2.南京师范大学 文学院，江苏省 南京市 210097； 3.南京师范大学 国际文化教育学院，江苏省 南京市 210097； *通讯作者，Email：wgqu nj@163.com) 摘 摘 摘要 要 要 汉语词语的离合现象是汉语中一种词语可分可合的特殊现象。本文采用字符级序列标 注方法解决二字动词离合现象的自动识别问题，以避免中文分词及词性标注的错误传 递，节省制定匹配规则与特征模板的人工开支。在训练过程中微调BERT中文预训练 模型，获取面向目标任务的字符向量表示，并引入掩码机制对模型隐藏离用法中分离 的词语，减轻词语本身对识别结果的影响，强化中间插入成分的学习，并对前后语素 采用不同的掩码以强调其出现顺序，进而使模型具备了识别复杂及偶发性离用法的能 力。为获得含有上下文信息的句子表达，将原始的句子表达与采用掩码的句子表达分 别输入两个不同参数的BiLSTM层进行训练，最后采用CRF算法捕捉句子标签序列的 依赖关系。本文提出的BERT MASK + 2BiLSTMs + CRF模型比现有最优的离合词 识别模型提高了2.85%的F1 值。 关 关 关键 键 键词 词 词： ： ： 离合词 ；自动识别 ；掩码机制 ；神经网络 Research on Recognition of the Separation and Reunion Phenomena of Words in Chinese ZHOU Lu1, QU Weiguang1,2, WEI Tingxin2,3, ZHOU Junsheng1, LI Bin2, GU Yanhui1 (1.School of Computer and Electronic Information/School of Artificial Intelligence, Nanjing Normal University, Nanjing, Jiangsu 210023, China; 2.School of Chinese Language and Literature, Nanjing Normal University, Nanjing, Jiangsu 210097, China; 3.International College for Chinese Studies, Nanjing Normal University, Nanjing, Jiangsu 210097, China; *Corresponding, Email: wgqu nj@163.com) Abstract The Separation of words where characters of a word can be used separately is the special phenomena in Chinese. This paper proposes a character-level sequence tagging method for automatic recognition of separable two-character verbs, which avoids the error propagation of Chinese word segmentation and part of speech tagging, and saves the labor cost of making rules and feature templates. In the process of training, the Chinese pre-trained model BERT is fine-tuned to obtain the vector representations of characters for the target task. Meanwhile, the MASK mechanism is introduced, where the separable words within the separable structures is hidden from the model, to reduce the influence of words themselves on the recognition results and strengthen the learning of the middle insertions in separable structures, and further, different masks for the ©2021 中国计算语言学大会 根据《Creative Commons Attribution 4.0 International License》许可出版 基金项目：国家自然科学基金“汉语抽象意义表示关键技术研究”(61772278)；国家自然科学基金“向量组合学习框架 下基于依存混合树的中文语义解析研究”(61472191)；江苏省高校哲学社会科学基金“面向机器学习的汉语复句语料 建设研究”(2019JSA0220)；国家社会科学基金“中文抽象语义库的构建及自动分析研究”（18BYY127）。 CCL 2021 第 二 十 届 中 国 计 算 语 言 学 大 会 论 文 集 ， 第 97页 -第 107页 ， 呼 和 浩 特 ， 中 国 ， 2021年 8月 13日 至 15日 。 (c) 2021 中 国 中 文 信 息 学 会 计 算 语 言 学 专 业 委 员 会 计算语言学 front and back morphemes are used to emphasize their occurrence complicated and occurrence order. Therefore, such implements give the model the ability to recognize complicated and occasional separation forms. To obtain the sentence representation with context information, the original sentence representation and the mask sentence representation are input into two BiLSTM modules with two different parameters for training, respectively. Finally, the CRF algorithm is used to capture the dependency of the tag sequence of the sentence of the sentence. The proposed BERT MASK + 2BiLSTMs + CRF model improves the F1 value by 2.85% compared with the existing optimal separable word recognition model. Keywords: Separable Words , Automatic Recognition …
Content truncated to 500 words
Labeled Text
AbstractIn the paper we present a ‘pre-training’+‘post-training’+‘fine-tuning’ three-stage paradigm which is a supplementary framework for the standard ‘pre-training’+‘fine-tuning’ languagemodel approach. Furthermore based on three-stage paradigm we present a language modelnamed PPBERT. Compared with original BERT architecture that is based on the standard two-stage paradigm we do not fine-tune pre-trained model directly but rather post-train it on the domain or task related dataset first which helps to better incorporate task-awareness knowl-edge and domain-awareness knowledge within pre-trained model also from the training datasetreduce bias. Extensive experimental results indicate that proposed model improves the perfor-mance of the baselines on 24 NLP tasks which includes eight GLUE benchmarks eight Su-perGLUE benchmarks six extractive question answering benchmarks. More remarkably our proposed model is a more flexible and pluggable model where post-training approach is able to be plugged into other PLMs that are based on BERT. Extensive ablations further validate the effectiveness and its state-of-the-art (SOTA) performance. The open source code pre-trained models and post-trained models are available publicly.
Publication Details
- Paper ID:
- 3590
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan