Single Model Ensemble using Pseudo-Tags and Distinct Vectors

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3006–3013 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3006 Single Model Ensemble using Pseudo-Tags and Distinct Vectors Ryosuke Kuwabara1, Jun Suzuki23, Hideki Nakayama1 1 The University of Tokyo, 2 Tohoku University, 3 RIKEN {kuwabara, nakayama}@nlab.ci.i.u-tokyo.ac.jp jun.suzuki@ecei.tohoku.ac.jp Abstract Model ensemble techniques often increase task performance in neural networks; however, they require increased time, memory, and man- agement effort. In this study, we propose a novel method that replicates the effects of a model ensemble with a single model. Our ap- proach creates K-virtual models within a sin- gle parameter space using K-distinct pseudo- tags and K-distinct vectors. Experiments on text classification and sequence labeling tasks on several datasets demonstrate that our method emulates or outperforms a traditional model ensemble with 1/K-times fewer param- eters. 1 Introduction A model ensemble is a promising technique for increasing the performance of neural network mod- els (Lars. and Peter., 1990; Anders and Jesper, 1994). This method combines the outputs of multi- ple models that are individually trained using the same training data. Recent submissions to natural language processing(NLP) competitions are primar- ily composed of neural network ensembles (Bojar et al., 2018; Barrault et al., 2019). Despite its ef- fectiveness, a model ensemble is costly. Because it handles multiple models, it requires increased time for training and inference, increased memory, and greater management effort. Therefore, the model ensemble technique cannot always be applied to real systems, as many systems, such as edge de- vices, must work with limited computational re- sources. In this study, we propose a novel method that replicates the effects of the ensemble technique with a single model. Following the principle that aggregating multiple models improves per- formance, we create multiple virtual models in a shared space. Our method virtually inflates the training data K times with K-distinct pseudo-tags [Tag 1] I watched this .. [Tag 2] I watched this .. [Tag 3] I watched this .. 𝒐𝟑 𝒆𝟎:𝑻 𝟑 '𝒆𝟎:𝑻 𝟑 𝜱(𝑬𝑵𝑪('𝒆𝟎:𝑻 𝟏 )) 𝜱(𝑬𝑵𝑪('𝒆𝟎:𝑻 𝟐 )) 𝜱 (𝑬𝑵𝑪('𝒆𝟎:𝑻 𝟑 )) Aggregate 𝒐𝟏 𝒆𝟎:𝑻 𝟏 '𝒆𝟎:𝑻 𝟏 𝒐𝟐 𝒆𝟎:𝑻 𝟐 '𝒆𝟎:𝑻 𝟐 Figure 1: Overview of our proposed method. A single model processes the same input with distinct pseudo- tags. Each pseudo-tag defines the k-th virtual model, and the corresponding vector ok is added to the em- bedding. Thus, the model function of a singe model φ (ENC(·)) generates different outputs. appended to all input data. It also incorporates K- distinct vectors, which correspond to pseudo-tags. Each pseudo-tag k ∈ {1, . . . , K} is attached to the beginning of the input sentence, and the k-th vector is added to the embedding vectors for all tokens in the input sentence. Fig. 1 presents a brief overview of our proposed method. Intuitively, this opera- tion allows the model to shift the embedding of the same data to the k-th designated subspace and can be interpreted as explicitly creating K virtual mod- els in a shared …

Content truncated to 500 words