MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6131 - 6142 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators Zhixing Tan1,3,4, Xiangwen Zhang6, Shuo Wang1,3,4, and Yang Liu1,2,3,4,5 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Beijing National Research Center for Information Science and Technology 5International Innovation Center of Tsinghua University, Shanghai, China 6Kuaishou Tech, Co. Abstract Prompting has recently been shown as a promis- ing approach for applying pre-trained language models to perform downstream tasks. We present Multi-Stage Prompting, a simple and automatic approach for leveraging pre-trained language models to translation tasks. To better mitigate the discrepancy between pre-training and translation, MSP divides the translation process via pre-trained language models into multiple separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre- trained language models better shift to trans- lation tasks. We conduct extensive experi- ments on three translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained lan- guage models. 1 1 Introduction Prompting (Brown et al., 2020; Lester et al., 2021), which refers to the approach of generating task- specific outputs from language models (LMs) by conditioning on extra information (known as prompts), has emerged as a new way of using LMs to perform natural language processing (NLP) tasks (Gao et al., 2020; Liu et al., 2021). While being efficient in parameters (Lester et al., 2021), prompting can enable mixed-task inference, which is not possible for other related approaches like finetuning or adapter-based tuning (Li and Liang, 2021; Lester et al., 2021). Prompting also opens the possibility of using a single pre-trained LM to perform all NLP tasks (Liu et al., 2021). Machine translation (MT), which involves trans- formations between two languages, is considered one of the most challenging tasks in NLP (Koehn Corresponding to: Z. Tan (zxtan@tsinghua.edu.cn) and Y. Liu (liuyang2011@tsinghua.edu.cn) 1Source code is available at https://github.com/ THUNLP-MT/PLM4MT. and Knowles, 2017). While neural machine trans- lation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) is the current de facto approach for machine translation, using pre-trained LMs as translators via prompting is ap- pealing in several aspects. For example, for the method described in this paper, supporting a new translation direction with a pre-trained LM occu- pies disk spaces below 20M, which is much smaller than training a separate neural machine translation model, where the model size is typically larger than 60M per language pair for the Transformer archi- tecture. 2 Furthermore, the pre-trained LM also re- tains the ability to perform other downstream tasks, which is an important characteristic that has not been validated available on neural machine transla- tion models. However, it is challenging to leverage pre-trained LMs to translation tasks via prompting. …

Content truncated to 500 words