DeepSPIN: Deep Structured Prediction for Natural Language Processing

DeepSPIN: Deep Structured Prediction for Natural Language Processing Andr´e F. T. Martins, Vlad Niculae, Erick Fonseca, Ben Peters, Gonc¸alo Correia, Tsvetomila Mihaylova, Marcos Treviso, Pedro Martins Instituto de Telecomunicac¸˜oes and Unbabel, Lisbon, Portugal andre.t.martins@tecnico.ulisboa.pt Abstract DeepSPIN1 is a research project funded by the European Research Council (ERC) whose goal is to develop new neural struc- tured prediction methods, models, and al- gorithms for improving the quality, inter- pretability, and data-efficiency of natural language processing (NLP) systems, with special emphasis on machine translation and quality estimation applications. 1 Description Neural network models became the standard in NLP applications, with impressive results in ma- chine translation (Bahdanau et al., 2015; Vaswani et al., 2017). New language interfaces (digital as- sistants, customer service bots) are emerging as the next technologies for seamless, multilingual com- munication among humans and machines. From a machine learning perspective, many problems in NLP can be characterized as structured prediction: they involve predicting structurally rich and inter- dependent outputs. In spite of this, current neural NLP systems ignore the structural complexity of human language, relying on simplistic and error- prone greedy search procedures. This leads to crit- ical mistakes in MT, such as words being dropped or named entities mistranslated. The DeepSPIN project attacks these fundamen- tal problems by bringing together deep learning and structured prediction. This is done in three fronts: better generation strategies, beyond left-to- right search; induction of sparse latent structure to c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1Project website: https://deep-spin.github.io. make networks more interpretable; and incorpora- tion of weak supervision to reduce the need for labeled data. We focus here on the applications to machine translation, including some results that have already been obtained in the project. Alternate Generation Strategies. In Peters et al. (2019), we introduced sparse sequence-to- sequence models, with encouraging results in MT. These sparse losses endow MT systems with a small set of choices for the next word to be gen- erated. When the model is fully confident, it auto- completes longer phrases (Figure 1). This property is appealing for building interactive MT systems. A current problem with left-to-right decoders is exposure bias (MT systems not exposed to their own predictions at training time). To mitigate this problem, we proposed a new scheduled sampling technique to avoid teacher forcing in transform- ers Mihaylova and Martins (2019). Future work will look at alternate generation strategies for MT, inspired by preliminary work in sequence tagging (Martins and Kreutzer, 2017). Sparse Attention and Interpretability. One big goal of the DeepSPIN project is to make neural networks amenable to interpretation by humans. This is particularly useful in a scenario mixing MT and human post-editing. Our recent work on sparse and structured attention (Martins and Astudillo, 2016; Niculae et al., 2018) presents a promising avenue for enhancing interpretability (see Figure 1 for sparse word alignments), and we built on this idea in two directions: to re- duce repetitions in neural MT by …

Content truncated to 500 words