Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7465–7471 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7465 Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization Henry Elder ADAPT Centre Dublin City University henry.elder@adaptcentre.ie Robert Burke sharpobject@gmail.com Alexander O’Connor Autodesk, inc. alex.oconnor@autodesk.com Jennifer Foster School of Computing Dublin City University jennifer.foster@dcu.ie Abstract The Surface Realization Shared Tasks of 2018 and 2019 were Natural Language Genera- tion shared tasks with the goal of explor- ing approaches to surface realization from Universal-Dependency-like trees to surface strings for several languages. In the 2018 shared task there was very little difference in the absolute performance of systems trained with and without additional, synthetically cre- ated data, and a new rule prohibiting the use of synthetic data was introduced for the 2019 shared task. Contrary to the findings of the 2018 shared task, we show, in experiments on the English 2018 dataset, that the use of synthetic data can have a substantial positive effect – an improvement of almost 8 BLEU points for a previously state-of-the-art system. We analyse the effects of synthetic data, and we argue that its use should be encouraged rather than prohibited so that future research efforts continue to explore systems that can take advantage of such data. 1 Introduction The shallow task of the recent surface realization (SR) shared tasks (Belz et al., 2011; Mille et al., 2018, 2019) appears to be a relatively straightfor- ward problem. Given a tree of lemmas, a system has to restore the original word order of the sen- tence and inflect its lemmas, see Figure 1. Yet SR systems often struggle, even for a relatively fixed word order language such as English. Improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (NLG) sys- tems (Moryossef et al., 2019; Elder et al., 2019; come story AP : this the from This story comes from the AP: Figure 1: Example tree and reference sentence Castro Ferreira et al., 2019). In this paper we explore the use of synthetic data for the English shallow task. Synthetic data is created by taking an unlabelled sentence, pars- ing it with an open source universal dependency parser1 and transforming the result into the input representation. Unlike in the 2018 shared task, where a system trained with synthetic data performed roughly the same as a system trained on the original dataset (Elder and Hokamp, 2018; King and White, 2018), we find its use leads to a large improvement in performance. The state-of-the-art on the dataset is 72.7 BLEU-4 score (Yu et al., 2019b) – our system achieves a similar result of 72.3, which improves to 80.1 with the use of synthetic data. We anal- yse the ways in which synthetic data helps to im- prove performance, finding …

Content truncated to 500 words