DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 416–428 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 416 Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity Po-Nien Kung Yi-Cheng Chen Sheng-Siang Yin Tse-Hsuan Yang Department of Computer Science and Information EngineeringYun-Nung Chen National Taiwan University, Taipei, Taiwan {b06902012, b06902011, b06902103, b06902032}@csie.ntu.edu.tw y.v.chen@ieee.org Abstract Multi-task auxiliary learning utilizes a set of relevant auxiliary tasks to improve the perfor- mance of a primary task. A common usage is to manually select multiple auxiliary tasks for multi-task learning on all data, which raises two issues: (1) selecting beneficial auxiliary tasks for a primary task is nontrivial; (2) when the auxiliary datasets are large, training on all data becomes time-expensive and imprac- tical. Therefore, this paper focuses on ad- dressing these problems and proposes a time- efficient sampling method to select the data that is most relevant to the primary task. The proposed method allows us to only train on the most beneficial sub-datasets from the auxil- iary tasks, achieving efficient multi-task auxil- iary learning. The experiments on three bench- mark datasets (RTE, MRPC, STS-B) show that our method significantly outperforms random sampling and ST-DNN. Also, by applying our method, the model can surpass fully-trained MT-DNN on RTE, MRPC, STS-B, using only 50%, 66%, and 1% of data, respectively.1 1 Introduction In recent years, language model pre-training has achieved great success in almost all NLP fields (De- vlin et al., 2019; Lan et al., 2019; Liu et al., 2019c; Lewis et al., 2020; Radford et al.; Yang et al., 2019). By learning from large corpus text segments with- out supervision, the models are able to learn the general representation of word tokens and can be further fine-tuned on downstream tasks. Moreover, many downstream tasks have their related tasks, which may benefit from the shared information in the training signal. To better utilize the shared knowledge, multi-task learning (MTL) is a com- mon technique. The recent work (Liu et al., 2019a; Raffel et al., 2020; Aghajanyan et al., 2021) fo- cused on capturing the shared knowledge by learn- 1The source code is available at: https://github. com/MiuLab/FastMTL/. ing multiple tasks simultaneously between the pre- training and fine-tuning stage, in order to benefit the downstream tasks. Even though the massive MTL scheme is demonstrated to achieve the im- provement in terms of performance, it is either time or computing costly; for example, Aghajanyan et al. (2021) used over 4.8 million total labeled examples for MTL. On the other hand, in numerous cases, MTL is applied but we only aim at a single task per- formance. It is usually addressed as multi-task auxiliary learning, which targets to introduce aux- iliary tasks and datasets to boost the performance of the primary task (Chen et al., 2018; Du et al., 2018; Guo et al., 2019). In this scenario, how to wisely select the auxiliary tasks plays the most important role. One straightforward method is to select the auxiliary tasks according to their relat- …

Content truncated to 500 words