AVA: an Automatic eValuation Approach for Question Answering Systems

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5223–5233 June 6–11, 2021. ©2021 Association for Computational Linguistics 5223 AVA: an Automatic eValuation Approach for Question Answering Systems Thuy Vu Amazon Alexa AI Manhattan Beach, CA, USA thuyvu@amazon.com Alessandro Moschitti Amazon Alexa AI Manhattan Beach, CA, USA amosch@amazon.com Abstract We introduce AVA, an automatic evaluation ap- proach for Question Answering, which given a set of questions associated with Gold Stan- dard answers (references), can estimate system Accuracy. AVA uses Transformer-based lan- guage models to encode question, answer, and reference texts. This allows for effectively as- sessing answer correctness using similarity be- tween the reference and an automatic answer, biased towards the question semantics. To de- sign, train, and test AVA, we built multiple large training, development, and test sets on public and industrial benchmarks. Our innova- tive solutions achieve up to 74.7% F1 score in predicting human judgment for single answers. Additionally, AVA can be used to evaluate the overall system Accuracy with an error lower than 7% at 95% of confidence when measured on several QA systems. 1 Introduction Accuracy evaluation is essential both to guide sys- tem development as well as to estimate its quality, which is important for researchers, developers, and users. This is often conducted using benchmark datasets containing a data sample, possibly repre- sentative of the target data distribution, provided with Gold Standard (GS) labels (typically produced with a human annotation process). The evaluation is done by comparing the system output with the expected labels using some metrics. This approach falls short when the system out- put spans a large, possibly infinite set of correct items. For example, in retrieval-based Question Answering (QA) systems, a correct answer can be any string in the referent text database. For exam- ple, for the question, When did Marlins start?, an answer could be: The Miami Marlins began play in the 1993 season as the Florida Marlins; They started in 1993; They firstly played in 1993; In 1993; or any possible natural language text convey- ing the information that they started in 1993. As annotating all possible system pieces of output is infeasible, the standard approach is to re-evaluate the new output of the system manually. This dra- matically limits the experimentation velocity while significantly increases the development costs. A viable solution for specific NLP tasks such as Machine Translation (MT), automatically esti- mates an evaluation score between the system and the reference answers, which correlates with hu- man judgment, e.g., the BLEU score is one popular measure (Papineni et al., 2002). Such methods cannot be applied to a standard QA setting, since QA systems, e.g., those developed for TREC-QA track (Voorhees and Tice, 1999), have the purpose to provide correct answers and are evaluated with Accuracy, i.e., the percentage of correct answers. Segment overlapping metrics such as BLEU, ME- TEOR, or ROUGE do not provide a binary out- come, i.e., correct or incorrect (as this is not …

Content truncated to 500 words