Content Type Profiling of Data-to-Text Generation Datasets

Proceedings of the 29th International Conference on Computational Linguistics, pages 5770–5782 October 12–17, 2022. 5770 Content Type Profiling of Data-to-Text Generation Datasets Ashish Upadhyay and Stewart Massie School of Computing Robert Gordon University Aberdeen, UK {a.upadhyay,s.massie}@rgu.ac.uk Abstract Data-to-Text Generation (D2T) problems can be considered as a stream of time-stamped events with a text summary being produced for each. The problem becomes more challeng- ing when event summaries contain complex insights derived from multiple records either within an event, or across several events from the event stream. It is important to understand the different types of content present in the sum- mary to help us better define the system require- ments so that we can build better systems. In this paper, we propose a novel typology of con- tent types, that we use to classify the contents of event summaries. Using the typology, a pro- file of a dataset is generated as the distribution of the aggregated content types which captures the specific characteristics of the dataset and gives a measure of the complexity present in the problem. Through extensive experiments on different D2T datasets we demonstrate that neural generative systems specifically struggle to generate contents of complex types, high- lighting the need for improved D2T techniques. 1 Introduction An ecologically valid task requires the automated systems to resemble the real-world scenario as closely as possible in its output (de Vries et al., 2020). Accordingly, a Data-to-Text Generation (D2T) system needs to convey important insights extracted from the data in the textual summaries (Reiter, 2007; Gatt and Krahmer, 2018). Most D2T problems can be seen as a stream of time-stamped events with a textual summary of each event pre- senting the insights. An event is the time-period of interest for which the textual summary is written. For example, in sports reporting - a game played between two teams can be an event; whereas in weather forecasting - the time period and location for which the forecast is written can be considered one full event. The summaries can contain different types of facts with information sometimes coming The Bucks (10-10) handled the Heat (9-9) 109-85 on Friday night in Milwaukee. It was the second victory over Miami for the Bucks this season after emerging victorious in Miami 91-84 on Nov. 16. Milwaukee fell behind early but clawed back into the game in the second quarter and held a four-point advantage at half. The Bucks were led by an unlikely face in Kendall Marshall, who scored a season-high 20 points (7-8 FG, 4-5 3Pt) in 24 minutes. Figure 1: Part of a basketball summary showing dif- ferent types of content. Information in “bold” such as Bucks’ points in the game (109) can be directly copied from input data, while the ones in “italics” needs to be derived from multiple records such as the fact - Kendall Marshall leading the Bucks team. Finally, the infor- mation in “bold & italics” such as ‘this was Miami’s second victory against Bucks’ are derived using records from …

Content truncated to 500 words