An Empirical Exploration of Local Ordering Pre-training for Structured Prediction

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1770–1783 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1770 An Empirical Exploration of Local Ordering Pre-training for Structured Prediction Zhisong Zhang, Xiang Kong, Lori Levin, Eduard Hovy Language Technologies Institute, Carnegie Mellon University {zhisongz,xiangk,lsl,hovy}@cs.cmu.edu Abstract Recently, pre-training contextualized encoders with language model (LM) objectives has been shown an effective semi-supervised method for structured prediction. In this work, we empirically explore an alternative pre-training method for contextualized encoders. Instead of predicting words in LMs, we “mask out” and predict word order information, with a lo- cal ordering strategy and word-selecting ob- jectives. With evaluations on three typical structured prediction tasks (dependency pars- ing, POS tagging, and NER) over four lan- guages (English, Finnish, Czech, and Italian), we show that our method is consistently bene- ficial. We further conduct detailed error analy- sis, including one that examines a specific type of parsing error where the head is misidenti- fied. The results show that pre-trained con- textual encoders can bring improvements in a structured way, suggesting that they may be able to capture higher-order patterns and fea- ture combinations from unlabeled data. 1 Introduction Recently, pre-trained contextualized encoders (Pe- ters et al., 2018; Radford et al., 2019; Devlin et al., 2019) have been shown to be beneficial for NLP tasks, including structured prediction (Kulmizev et al., 2019; Kondratyuk and Straka, 2019). Most of the pre-training objectives are based on variants of language models (LM), that is, the model is trained to predict lexical items with partial inputs. Masked Language Model (MaskLM) is a typical example, popularized by BERT (Devlin et al., 2019), which masks out lexical tokens in the input sequences and predicts their identities. Since natural sentences contain not only lexical tokens but also their lin- earized word orders, it is a natural question if we can perform pre-training by “masking out” and re- covering word order information. Word order is an important method of grammat- ical encoding (Dryer, 2007), and can play an im- portant role in predicting basic sentence structures (Naseem et al., 2012; T¨ackstr¨om et al., 2013; Am- mar et al., 2016; Ahmad et al., 2019). Recently, Wang et al. (2018) pre-train an explicit word re- ordering model and show that its contextualized representations improve dependency parsing. In this work, we explore a local ordering pre- training strategy with word-selection objectives. Instead of completely discarding original word or- der information, we segment the input sentence into local bags of words and keep the ordering of these bags. Inside each bag, we discard all the local word orders and train the model to recover them. Furthermore, we simplify the training objec- tives: instead of training explicit word linearizers which require extra unidirectional decoders, we only ask the model to select original neighboring words. This scheme simplifies the pre-training pro- cedure and enhances the encoder since it can take information from the whole sentence. A similar idea is explored in StructBERT (Wang et al., 2020), which adopts a …

Content truncated to 500 words