Towards Automating Healthcare Question Answering in a Noisy Multilingual Low-Resource Setting

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 948–953 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 948 Towards Automating Healthcare Question Answering in a Noisy Multilingual Low-Resource Setting Jeanne E. Daniel Applied Mathematics Stellenbosch University jeanne.e.daniel@gmail.com Willie Brink Applied Mathematics Stellenbosch University wbrink@sun.ac.za Ryan Eloff Electrical & Electronic Engineering Stellenbosch University ryan.peter.eloff@gmail.com Charles Copley Data Science Praekelt Foundation charles@praekelt.org Abstract We discuss ongoing work into automating a multilingual digital helpdesk service available via text messaging to pregnant and breastfeed- ing mothers in South Africa. Our anonymized dataset consists of short informal questions, often in low-resource languages, with un- reliable language labels, spelling errors and code-mixing, as well as template answers with some inconsistencies. We explore cross- lingual word embeddings, and train paramet- ric and non-parametric models on 90K sam- ples for answer selection from a set of 126 templates. Preliminary results indicate that LSTMs trained end-to-end perform best, with a test accuracy of 62.13% and a recall@5 of 89.56%, and demonstrate that we can acceler- ate response time by several orders of magni- tude. 1 Introduction MomConnect is a project led by South Africa’s National Department of Health (NDOH), and is freely available at public clinics to all mothers and pregnant women who wish to sign up. The plat- form provides maternal support through text mes- saging in all 11 official languages of South Africa and has had 2.6 million registrations since 2014. MomConnect is unique in the public health sec- tor, allowing users to pose questions to helpdesk staff who respond manually. The recent introduc- tion of WhatsApp as a channel additional to SMS has increased the volume of questions substan- tially. This presents a significant challenge for the staffing complement, and median response time is currently 20 hours. The templated-based nature of answers pro- vides an immediate opportunity for using com- putational linguistics to automate the response pipeline. In a similar study, Engelhard et al. (2018) Language Users Afrikaans 2,578 English 51,250 Ndebele 97 N. Sotho 3,400 S. Sotho 749 Swati 287 Language Users Tsonga 1,361 Tswana 948 Venda 1,105 Xhosa 6,821 Zulu 17,615 Total 86,211 Table 1: Unique users in dataset per sign-up language. evaluated the need for and feasibility of automated message triage to improve helpdesk responsive- ness to high-priority messages. During 2018 we worked with the NDOH to gain access to MomConnect data for research purposes. After rigorous ethical clearance and user privacy protection protocols, we obtained a static copy of about 230,000 raw textual question-answer pairs. The primary objective of this research is to inves- tigate ways in which the burden on helpdesk staff can be reduced, which could enable MomConnect to scale towards a wider reach and a more effec- tive service. One aspect of this is to automate the question answering process. A language label is recorded when a user signs up at the clinic, but users are free to ask questions in any language, which poses a significant chal- lenge …

Content truncated to 500 words