Pre-training Universal Language Representation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5122–5133 August 1–6, 2021. ©2021 Association for Computational Linguistics 5122 Pre-training Universal Language Representation Yian Li1,2,3, Hai Zhao1,2,3,∗ 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University liya19@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract Despite the well-developed cut-edge represen- tation learning for language, most language representation models usually focus on spe- cific levels of linguistic units. This work introduces universal language representation learning, i.e., embeddings of different levels of linguistic units or text with quite diverse lengths in a uniform vector space. We pro- pose the training objective MiSAD that utilizes meaningful n-grams extracted from large un- labeled corpus by a simple but effective algo- rithm for pre-trained language models. Then we empirically verify that well designed pre- training scheme may effectively yield univer- sal language representation, which will bring great convenience when handling multiple lay- ers of linguistic objects in a unified way. Es- pecially, our model achieves the highest accu- racy on analogy tasks in different language lev- els and significantly improves the performance on downstream tasks in the GLUE benchmark and a question answering dataset. 1 Introduction In this paper, we propose universal language repre- sentation (ULR) that uniformly embeds linguistic units in different hierarchies in the same vector space. A universal language representation model encodes linguistic units such as words, phrases or sentences into fixed-sized vectors and handles mul- tiple layers of linguistic objects in a unified way. ULR learning may offer a great convenience when confronted with sequences of different lengths, es- pecially in tasks such as Natural Language Under- standing (NLU) and Question Answering (QA), ∗ Corresponding author. This paper was partially sup- ported by National Key Research and Development Pro- gram of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), Huawei-SJTU long term AI project, Cutting- edge Machine Reading Comprehension and Language Model. This work was supported by Huawei Noah’s Ark Lab. hence it is of great importance in both scientific research and industrial applications. As is well known, embedding representation for a certain linguistic unit (i.e., word) en- ables linguistics-meaningful arithmetic calculation among different vectors, also known as word anal- ogy (Mikolov et al., 2013). For example: King − Man = Queen − Woman In fact, manipulating embeddings in the vector space reveals syntactic and semantic relations be- tween the original symbol sequences and this fea- ture is indeed useful in true applications. For ex- ample, “London is the capital of England” can be formulized as: England + capital ≈ London Then given two documents one of which contains “England” and “capital”, the other contains “Lon- don”, we consider them relevant. While a ULR model may generalize such good analogy features onto free text …

Content truncated to 500 words