Seq2SeqPy: A Lightweight and Customizable Toolkit for Neural Sequence-to-Sequence Modeling

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7140–7144 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7140 Seq2SeqPy: A Lightweight and Customizable Toolkit for Neural Sequence-to-Sequence Modeling Raheel Qader, Franc¸ois Portet, Cyril Labb´e Universit´e Grenoble Alpes, LIG Grenoble, France raheel.qader@univ-grenoble-alpes.fr {francois.portet, cyril.labbe}@imag.fr Abstract We present Seq2SeqPy a lightweight toolkit for sequence-to-sequence modeling that prioritizes simplicity and ability to customize the standard architectures easily. The toolkit supports several known models such as Recurrent Neural Networks, Pointer Generator Networks, and transformer model. We evaluate the toolkit on two datasets and we show that the toolkit performs similarly or even better than a very widely used sequence-to-sequence toolkit. Keywords: sequence-to-sequence modeling, deep learning, toolkit 1. Introduction Neural models have attracted a lot of attention in the past few years. They have become standard models in several tasks such as machine translation, summarization, Natural Language Generation (NLG), etc. Recently, several cor- porations and institutes have released frameworks to help the progress of neural networks. Two of the most widely adopted frameworks are Tensorflow by Google and Py- torch by Facebook AI. These frameworks provide enor- mous amount of advantages when it comes to designing and training neural networks. They handle very compli- cated tasks such as automatic differentiation and comput- ing with GPU which makes working with neural networks much easier. In addition, several toolkits have been developed on top of these frameworks to help researchers quickly perform experiments on machine learning tasks in general and se- quence modeling in particular. Among other, some of the most known toolkits are: OpenNMT (Klein et al., 2017), Fairseq (Ott et al., 2019) and Sockeye (Hieber et al., 2017). Most of these toolkits have extremely complicated cod- ing structures and they are mainly targeting standard tasks like machine translation with very large corpora such as WMT1. This complexity in the code is the artifact of sup- porting a very large number of features that are needed in certain cases. However, there are cases where all these features might not be necessary, for instance, researcher who are new to sequence-to-sequence (seq2seq) modeling might need more simpler codes to start with. In addition, there might be cases where we need to customize the stan- dard seq2seq architectures to add own research goals. In such cases, due to their complexity, changing the aforemen- tioned toolkits is very difficult and time consuming. In this paper we present Seq2SeqPy, a Pytorch2 based seq2seq modeling toolkit that we have built by prioritizing simplicity and ability to customize the provided architec- tures easily. The main target of this toolkit is researchers and students who are new to deep learning and seq2seq 1http://www.statmt.org/wmt15 2https://pytorch.org/ models. The main features of Seq2SeqPy are: • Support for several seq2eq models • Easy to use command line arguments • Implementation in Python based on PyTorch • Ability to customize and add new architectures • Ability to easily add pre-trained language models such as BERT • Easy to track …

Content truncated to 500 words