Evaluation of Dataset Selection for Pre-Training and Fine-Tuning Transformer Language Models for Clinical Question Answering

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 531–539 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 531 Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions Eugénio Ribeiro1,2, Ricardo Ribeiro1,3, David Martins de Matos1,2 1 INESC-ID Lisboa, Portugal 2 Instituto Superior Técnico, Universidade de Lisboa, Portugal 3 Instituto Universitário de Lisboa (ISCTE-IUL), Portugal {eugenio.ribeiro, ricardo.ribeiro, david.matos}@inesc-id.pt Abstract ISO 24617-2, the ISO standard for dialog act annotation, sets the ground for more comparable research in the area. However, the amount of data annotated according to it is still reduced, which impairs the development of approaches for automatic recognition. In this paper, we describe a mapping of the original dialog act labels of the LEGO corpus, which have been neglected, into the communicative functions of the standard. Although this does not lead to a complete annotation according to the standard, the 347 dialogs provide a relevant amount of data that can be used in the development of automatic communicative function recognition approaches, which may lead to a wider adoption of the standard. Using the 17 English dialogs of the DialogBank as gold standard, our preliminary experiments have shown that including the mapped dialogs during the training phase leads to improved performance while recognizing communicative functions in the Task dimension. Keywords: dialog acts, iso standard, communicative functions, LEGO corpus 1. Introduction It is valuable for a dialog system to identify the intention behind its conversational partners’ words since it provides an important cue concerning the information contained in a segment and how it should be interpreted. According to Searle (1969), that intention is revealed by dialog acts, which are the minimal units of linguistic communication. Conse- quently, automatic dialog act recognition is an important task in the context of Natural Language Processing (NLP), which has been widely explored over the years. However, dialog act annotation has typically been performed in the context of projects or the development of datasets, each in- troducing new tag sets or modifying the existing ones. This led to a wide scattering of data in terms of annotation, which hardens the comparison of results and conclusions obtained using different automatic dialog act recognition approaches. In an attempt to set the ground for more comparable research in the area, a standard for dialog act annotation was devel- oped (Bunt et al., 2012; Bunt et al., 2017). However, an- notating dialogs according to this standard is an exhaustive process, especially since the annotation does not consist of a single label, which in the standard nomenclature is called a communicative function, but rather of a complex structure which includes information regarding the semantic dimen- sion of the dialog act and relations with other segments, among others. Consequently, currently, the only publicly available annotated dialogs are those provided in the Di- alogBank (Bunt et al., 2016; Bunt et al., 2019) (Bunt et al., 2016). These are not enough for training models using the data-hungry deep learning approaches …

Content truncated to 500 words