SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4325–4333 June 6–11, 2021. ©2021 Association for Computational Linguistics 4325 SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain Ruochen Zhang and Carsten Eickhoff Brown University {ruochen_zhang, carsten}@brown.edu Abstract In the pursuit of natural language understand- ing, there has been a long standing interest in tracking state changes throughout narratives. Impressive progress has been made in model- ing the state of transaction-centric dialogues and procedural texts. However, this problem has been less intensively studied in the realm of general discourse where ground truth de- scriptions of states may be loosely defined and state changes are less densely distributed over utterances. This paper proposes to turn to simplified, fully observable systems that show some of these properties: Sports events. We curated 2,263 soccer matches including time- stamped natural language commentary accom- panied by discrete events such as a team scor- ing goals, switching players or being penalized with cards. We propose a new task formulation where, given paragraphs of commentary of a game at different timestamps, the system is asked to recognize the occurrence of in-game events. This domain allows for rich descrip- tions of state while avoiding the complexities of many other real-world settings. As an ini- tial point of performance measurement, we in- clude two baseline methods from the perspec- tives of sentence classification with temporal dependence and current state-of-the-art gener- ative model, respectively, and demonstrate that even sophisticated existing methods struggle on the state tracking task when the definition of state broadens or non-event chatter becomes prevalent. 1 Introduction State tracking, the task of maintaining explicit rep- resentations of user requests and agent responses, has long been a key component of dialogue systems (Williams et al., 2013; Henderson et al., 2014a,b; Kim et al., 2016). The same challenge arises during reading comprehension of procedural texts (recipes, how-to guides, etc.) where systems focus on pre- dicting changes of object attributes at the entity- level (a car window may transition from foggy to clear) (Dalvi et al., 2018; Tandon et al., 2020). However, both of these state tracking variants rely on transaction-based or turn-based data such as transactional dialogues or procedure descriptions that are information-dense. Few works have stud- ied state tracking tasks where state changes occur infrequently while a large proportion of messages are “chatter”. As an alternative to altogether unrestricted state tracking—a task that is daunting due to the com- plexity of even describing ground-truth states in a discrete manner—we resort to a simpler and more self-contained setting: sports competitions. Given the stream of natural language utterances with which a commentator describes the events in a real-world setting (here a sports competition), an ideal natural language understanding system would maintain and reason over a coherent and ac- curate representation of the match based on how the commentator described it. This representation can, in turn, be used for downstream tasks such as inference or language generation. …

Content truncated to 500 words