Evaluating Recognizing Question Entailment Methods for a Portuguese Community Question-Answering System about Diabetes Mellitus

Proceedings of Recent Advances in Natural Language Processing, pages 234–243 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_028 234 Evaluating Recognizing Question Entailment Methods for a Portuguese Community Question-Answering System about Diabetes Mellitus Thiago Castro Ferreira1 Jo˜ao Victor de Pinho Costa1 Isabela Rigotto1 Vitoria Portella1 Gabriel Frota1 Ana Luisa A. R. Guimar˜aes1 Adalberto Penna1 Isabela Lee1 Tayane A. Soares1 Sophia Rolim1 Rossana Cunha1 Celso Franc¸a1 Ariel Santos1 Rivaney F. Oliveira1 Abisague Langbehn2 Daniel Hasan Dalip3 Marcos Andr´e Gonc¸alves1 Rodrigo Bastos F´oscolo1 and Adriana Pagano1 1Federal University of Minas Gerais (UFMG), Brazil 2Technical High School of UFMG, Brazil 3Federal Center for Technological Education of Minas Gerais (CEFET-MG), Brazil thiagocf05@ufmg.br Abstract This study describes the development of a Portuguese Community-Question Answering benchmark in the domain of Diabetes Melli- tus using a Recognizing Question Entailment (RQE) approach. Given a premise question, RQE aims to retrieve semantically similar, al- ready answered, archived questions. We build a new Portuguese benchmark corpus with 785 pairs between premise questions and archived answered questions marked with relevance judgments by medical experts. Based on the benchmark corpus, we leveraged and evalu- ated several RQE approaches ranging from traditional information retrieval methods to novel large pre-trained language models and ensemble techniques using learn-to-rank ap- proaches. Our experimental results show that a supervised transformer-based method trained with multiple languages and for multiple tasks (MUSE) outperforms the alternatives. Our results also show that ensembles of methods (stacking) as well as a traditional (light) in- formation retrieval method (BM25) can pro- duce competitive results. Finally, among the tested strategies, those that exploit only the question (not the answer), provide the best effectiveness-efficiency trade-off. Code is pub- licly available1. 1 Introduction Question answering (QA) aims to automatically re- trieve precise, rather than merely relevant, answers to a given question. The field has faced exponential progress along the years with new corpora (Ra- jpurkar et al., 2016; Ahmad et al., 2019) as well as computational models which approach the task from different perspectives. One of these is known as Recognizing Question Entailment (RQE). Given a premise question (aka query), a RQE approach aims to retrieve semantically similar 1https://github.com/Dia-Bete/RANLP2021 archived questions which have been already an- swered (Ben Abacha and Demner-Fushman, 2019). The task became relevant thanks to popular Com- munity QA forums, such as Yahoo Answers, Quora and Stack Overflow, where an RQE approach is used to automatically search a large body of mate- rial looking up for archived question-answer pairs entailing a user posed question. Besides popular forums, a domain in which RQE approaches are highly beneficial and have been ex- tensively studied is the medical one. Russell-Rose and Chamberlain (2017) showed that, when using traditional information retrieval search engines to query medical information, healthcare profession- als spend on average 60 minutes to formulate a search strategy, 3 minutes to analyse the relevance of a retrieved document, and 4 hours of total search time. Ben Abacha and Demner-Fushman (2019) suggest that healthcare consumers may also benefit from QA systems through which they can ask for desired information in …

Content truncated to 500 words