AIA-BDE: A Corpus of FAQs in Portuguese and their Variations

14170

Authors

nan

Abstract

AbstractWe present AIA-BDE, a corpus of 380 domain-oriented FAQs in Portuguese and their variations, i.e., paraphrases or entailed questions, created manually, by humans, or automatically, with Google Translate. Its aims to be used as a benchmark for FAQ retrieval and automatic question-answering, but may be useful in other contexts, such as the development of task-oriented dialogue systems, or models for natural language inference in an interrogative context. We also report on two experiments. Matching variations with their original questions was not trivial with a set of unsupervised baselines, especially for manually created variations. Besides high performances obtained with ELMo and BERT embeddings, an Information Retrieval system was surprisingly competitive when considering only the first hit. In the second experiment, text classifiers were trained with the original questions, and tested when assigning each variation to one of three possible sources, or assigning them as out-of-domain. Here, the difference between manual and automatic variations was not so significant.

Keywords

nan

Full Content

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 523–530 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 523 PÂTÉ: A Corpus of Temporal Expressions for the In-car Voice Assistant Domain Alessandra Zarcone1, Touhidul Alam2,3, Zahra Kolagar2 Fraunhofer IIS (1ALabs, 2AME-S), 3Universität Stuttgart (IMS) Am Wolfsmantel 33, 91058 Erlangen, Germany, Pfaffenwaldring 5b, 70569 Stuttgart, Germany {alessandra.zarcone, alamtl, kolagaza}@iis.fraunhofer.de Abstract The recognition and automatic annotation of temporal expressions (e.g. Add an event for tomorrow evening at eight to my calendar) is a key module for AI voice assistants, in order to allow them to interact with apps (for example, a calendar app). However, in the NLP literature, research on temporal expressions has focused mostly on data from the news, from the clinical domain, and from social media. The voice assistant domain is very different than the typical domains that have been the focus of work on temporal expression identification, thus requiring a dedicated data collection. We present a crowdsourcing method for eliciting natural-language commands containing temporal expressions for an AI voice assistant, by using pictures and scenario descriptions. We annotated the elicited commands (480) as well as the commands in the Snips dataset following the TimeML/TIMEX3 annotation guidelines, reaching a total of 1188 annotated commands. The commands can be later used to train the NLU components of an AI voice assistant. Keywords: voice assistant, TIMEX3, TimeML, crowdsourcing 1. Introduction Voice assistants are becoming pervasive in our lives. In or- der to support their users with different tasks (for example, booking a table at a restaurant for a specific time), they rely on so-called Natural Language Understanding (NLU) com- ponents, such as intent classification or slot filling (Tur and De Mori, 2011). NLU components map natural-language commands (e.g. Call the restaurant) to abstract represen- tations of the user’s intentions (e.g. Call(Restaurant)) by relying on usage patterns, in order to trigger actions (e.g. responses, queries on a knowledge base, operation of home automation devices)1. NLU components typically use su- pervised methods and rely on large-scale labeled domain- specific datasets for training and testing (Tur et al., 2018), which should reflect the way people address voice assis- tants (Fraser and Gilbert, 1991). In order to help users with everyday scheduling tasks, AI voice assistants need to identify and extract temporal information from tempo- ral expressions (TEs, e.g. tomorrow at eight), for example to correctly set reminders, or help the user interact with a calendar. The automatic identification, tagging and normalization of temporal expressions has been a widely researched topic in NLP (Pustejovsky et al., 2009; UzZaman et al., 2013), and has typically focused on large text documents, for ex- ample in the news domain (e.g. TimeBank and AQUAINT, both used as gold standard in TempEval-3) or in the clini- cal domain (Styler IV et al., 2014), with the aim to iden- tify relations between events mentioned in the text and their chronological order. However, to our knowledge, no dataset with a rich annotation of TEs …

Content truncated to 500 words

Labeled Text

AbstractWe present AIA-BDE, a corpus of 380 domain-oriented FAQs in Portuguese and their variations, i.e., paraphrases or entailed questions, created manually, by humans, or automatically, with Google Translate. Its aims to be used as a benchmark for FAQ retrieval and automatic question-answering, but may be useful in other contexts, such as the development of task-oriented dialogue systems, or models for natural language inference in an interrogative context. We also report on two experiments. Matching variations with their original questions was not trivial with a set of unsupervised baselines, especially for manually created variations. Besides high performances obtained with ELMo and BERT embeddings, an Information Retrieval system was surprisingly competitive when considering only the first hit. In the second experiment, text classifiers were trained with the original questions, and tested when assigning each variation to one of three possible sources, or assigning them as out-of-domain. Here, the difference between manual and automatic variations was not so significant.

Publication Details

Paper ID:
14170
Year:
2020
Publication:
TIPSTER
Publication Type:
conf
DOI:
nan