RevCore: Review-Augmented Conversational Recommendation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 96–102 August 1–6, 2021. ©2021 Association for Computational Linguistics 96 More Parameters? No Thanks! Zeeshan Khan1, Kartheek Akella1, Vinay P. Namboodiri2, C V Jawahar1 1CVIT, IIIT-H 2University of Bath {zeeshank606,sukruthkartheek}@gmail.com vpn22@bath.ac.uk, jawahar@iiit.ac.in Abstract This work studies the long-standing prob- lems of model capacity and negative interfer- ence in multilingual neural machine transla- tion (MNMT). We use network pruning tech- niques and observe that pruning 50-70% of the parameters from a trained MNMT model results only in a 0.29-1.98 drop in the BLEU score. Suggesting that there exist large re- dundancies in MNMT models. These observa- tions motivate us to use the redundant parame- ters and counter the interference problem effi- ciently. We propose a novel adaptation strat- egy, where we iteratively prune and retrain the redundant parameters of an MNMT to im- prove bilingual representations while retain- ing the multilinguality. Negative interference severely affects high resource languages, and our method alleviates it without any additional adapter modules. Hence, we call it parameter- free adaptation strategy, paving way for the ef- ficient adaptation of MNMT. We demonstrate the effectiveness of our method on a 9 lan- guage MNMT trained on TED talks, and report an average improvement of +1.36 on high re- source pairs. Code will be released here. 1 Introduction Multilingual neural machine translation(MNMT) has seen various advances in recent years (Dong et al., 2015; Firat et al., 2016; Zoph et al., 2016; Tan et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019). However, the core principle behind the effectiveness in terms of modelling multiple languages remains the same, i.e., sharing all the model parameters between all the languages (John- son et al., 2017). Although highly scalable and effective, the performance on high resource lan- guages decreases as more low resource languages are added in the model; this is called negative in- terference. To overcome this, recent works (Bapna and Firat, 2019; Philip et al., 2020; Zhang et al., 2020) proposed language-specific adapter modules, which provide extra parameters to learn language specific representations, and overcomes the effect of negative interference caused by a high degree of parameter sharing. In this paper, we propose an alternative to adapter modules. Instead of adding more parame- ters, we show that the Transformer (Vaswani et al., 2017) has enough capacity to model multiple lan- guages and overcome negative interference effec- tively. Inspired by the work of Mallya and Lazeb- nik (2018), we apply iterative pruning to free up the redundant parameters from an MNMT, and retrain them to learn language specific representations. We start with a trained MNMT model, and prune a frac- tion of the model parameters, we freeze the surviv- ing parameters and retrain the free ones on a bilin- gual dataset. This process is iteratively applied for each bilingual pair to get bilingual masks over all the model parameters, as illustrated in figure 1. We show that using only a fraction of redundant param- eters, significantly improves the performance …

Content truncated to 500 words