Knowledge Graph - Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6260–6270 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6260 Knowledge Graph – Deep Learning: A Case Study in Question Answering in Aviation Safety Domain Ankush Agarwal1, Raj Gite1, Shreya Laddha1, Pushpak Bhattacharyya1, Satyanarayan Kar2, Asif Ekbal3, Prabhjit Thind2, Rajesh Zele1, Ravi Shankar2 1IIT Bombay, 2Honeywell, 3IIT Patna {ankushagrawal, rajgite, pb}@cse.iitb.ac.in, {shreyaladdha, rajeshzele}@ee.iitb.ac.in, asif@iitp.ac.in, {satya.kar, prabhjit.thind, ravishankar.r}@honeywell.com Abstract In the commercial aviation domain, there are a large number of documents, like accident reports of NTSB and ASRS, and regulatory directives ADs. There is a need for a system to efficiently access these diverse repositories to serve the demands of the aviation industry, such as maintenance, compliance, and safety. In this paper, we propose a Knowledge Graph (KG) guided Deep Learning (DL) based Question Answering (QA) system to cater to these requirements. We construct a KG from aircraft accident reports and contribute this resource to the community of researchers. The efficacy of this resource is tested and proved by the proposed QA system. Questions in Natural Language are converted into SPARQL (the interface language of the RDF graph database) queries and are answered from the KG. On the DL side, we examine two different QA models, BERT-QA and GPT3-QA, covering the two paradigms of answer formulation in QA. We evaluate our system on a set of handcrafted queries curated from the accident reports. Our hybrid KG + DL QA system, KGQA + BERT-QA, achieves 7% and 40.3% increase in accuracy over KGQA and BERT-QA systems respectively. Similarly, the other combined system, KGQA + GPT3-QA, achieves 29.3% and 9.3% increase in accuracy over KGQA and GPT3-QA systems respectively. Thus, we infer that the combination of KG and DL is better than either KG or DL individually for QA, at least in our chosen domain. Keywords: Question Answering, Knowledge Discovery/Representation, Information Extraction, Information Retrieval 1. Introduction An extensive set of documents is used in the aerospace sector, viz., system descriptions, manuals, and proce- dures. In addition to the industrial technical manuals, which have limited accessibility, there is a wide range of publicly available datasets related to aircraft safety. The majority of these datasets are subject to specific regulations or procedures, and safety professionals utilize them to investigate accidents and trends in avia- tion safety. Naturally, a Question Answering system is desired due to the large number and extensive length of such documents. For example, when investigating an accident, the ability to query the documents is required to locate similar events or find rare occurrences. In this paper, we present an Aviation Knowledge Graph, constructed from the accident reports, to help safety experts study the reported accidents. It contains detailed information about the accidents’ cause, effect, aircraft, pilot, and meteorological conditions. KGs provides a way to integrate and relate a large number of scattered, isolated pieces of information, thus convert- ing it into knowledge that supports making complex inferences. However, domain expertise is required to construct …

Content truncated to 500 words