An Error Analysis Framework for Shallow Surface Realization

An Error Analysis Framework for Shallow Surface Realization Anastasia Shimorina Yannick Parmentier Claire Gardent Universit´e de Lorraine, CNRS, LORIA, F-54000 Nancy, France {anastasia.shimorina, yannick.parmentier, claire.gardent}@loria.fr Abstract The metrics standardly used to evaluate Nat- ural Language Generation (NLG) models, such as BLEU or METEOR, fail to provide information on which linguistic factors impact performance. Focusing on Surface Realization (SR), the task of converting an unordered dependency tree into a well-formed sentence, we propose a framework for error analysis which permits identifying which features of the input affect the models’ results. This framework consists of two main components: (i) correlation analyses between a wide range of syntactic metrics and standard performance metrics and (ii) a set of techniques to automat- ically identify syntactic constructs that often co-occur with low performance scores. We demonstrate the advantages of our framework by performing error analysis on the results of 174 system runs submitted to the Multilingual SR shared tasks; we show that dependency edge accuracy correlate with automatic metrics thereby providing a more interpretable basis for evaluation; and we suggest ways in which our framework could be used to improve mod- els and data. The framework is available in the form of a toolkit which can be used both by campaign organizers to provide detailed, lin- guistically interpretable feedback on the state of the art in multilingual SR, and by individual researchers to improve models and datasets.1 1 Introduction Surface Realization (SR) is a naturallanguage gen- eration task that consists in converting a linguistic representation into a well-formed sentence. SR is a key module in pipeline generation mod- els, where it is usually the last item in a pipeline of modules designed to convert the input (knowledge graph, tabular data, numerical data) into a text. While end-to-end generation models have been 1Our code and settings to reproduce the experiments are available at https://gitlab.com/shimorina /tacl-2021. proposed that do away with such pipeline archi- tecture and therefore with SR, pipeline generation models (Duˇsek and Jurˇc´ıˇcek, 2016; Castro Ferreira et al., 2019; Elder et al., 2019; Moryossef et al., 2019) have been shown to perform on a par with these end-to-end models while providing increased controllability and interpretability (each step of the pipeline provides explicit interme- diate representations that can be examined and evaluated). As illustrated in, for example, Duˇsek and Jurˇc´ıˇcek (2016), Elder et al. (2019), and Li (2015), SR also has potential applications in tasks such as summarization and dialogue response generation. In such approaches, shallow dependency trees are viewed as intermediate structures used to mediate between input and output, and SR permits regen- erating a summary or a dialogue turn from these intermediate structures. Finally, multilingual SR is an important task in its own right in that it permits a detailed eval- uation of how neural models handle the varying word order and morphology of the different natu- ral languages. While neural language models are powerful at producing high quality text, the results of the multilingual SR tasks (Mille et al., 2018, 2019) clearly show …

Content truncated to 500 words