DeepSPIN: Deep Structured Prediction for Natural Language Processing

DeepSPIN: Deep Structured Prediction for Natural Language Processing André F. T. Martins, Ben Peters, Chrysoula Zerva, Chunchuan Lyu, Gonçalo Correia, Marcos Treviso, Pedro Martins, Tsvetomila Mihaylova Instituto de Telecomunicações and Unbabel, Lisbon, Portugal andre.t.martins@tecnico.ulisboa.pt Abstract DeepSPIN is a research project funded by the European Research Council (ERC), whose goal is to develop new neural struc- tured prediction methods, models, and al- gorithms for improving the quality, inter- pretability, and data-efficiency of natural language processing (NLP) systems, with special emphasis on machine translation and quality estimation. We describe in this paper the latest findings from this project. 1 Description The DeepSPIN project1 is an ERC Starting Grant (2019–2023) hosted at Instituto de Telecomuni- cações. Part of the work has been done in col- laboration with Unbabel, an SME in the crowd- sourcing translation industry. The main goal of DeepSPIN is to bring together deep learning and structured prediction techniques to solve struc- tured problems in NLP. The three main objectives are: developing better decoding strategies; making neural networks more interpretable through the in- duction of sparse structure; and incorporating of weak supervision to reduce the need for labeled data. We focus here on the applications to MT, in- cluding some of the recent results obtained in the project. Better Decoding Strategies. Our initial work on sparse sequence-to-sequence models (Peters et al., 2019) proposed a new class of decoders (called “entmax decoders”, shown in Fig. 1) which op- erate over a sparse probability distribution over © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1Project website: https://deep-spin.github.io. at on , This So And Here the tree of life . view look glimpse kind looking way vision gaze is another 92.9% 5.9% 1.3% <0.1% 49.8% 27.1% 19.9% 2.0% 0.9% 0.2% <0.1% <0.1% 95.7% 5.9% 1.3% Figure 1: Forced decoding using entmax for the German source sentence “Dies ist ein weiterer Blick auf den Baum des Lebens.” Only predictions with nonzero probability are shown at each time step. When consecutive predictions con- sist of a single word, we combine their borders to showcase auto-completion potential. words, which prunes hypotheses automatically. In (Peters and Martins, 2021), we have shown that entmax decoders are better calibrated and less prone to the length bias problem and developed a new label smoothing technique. We also pre- sented entmax sampling for text generation, with improved generation quality (Martins et al., 2020). Another line of work concerns modeling of context in machine translation. We introduced conditional cross-mutual information (CXMI), a technique to measure the effective use of contextual informa- tion by context-aware systems, and context-aware word dropout, which increases its use, leading to improvements (Fernandes et al., 2021). We also compared the models’ use of context to that of hu- mans for translating ambiguous words, using the latter as extra supervision (Yin et al., 2021). Sparse Attention and Explainability. A key objective of DeepSPIN is to make neural networks more interpretable to humans. Building upon our work on sparse …

Content truncated to 500 words