A Discrete Hard EM Approach for Weakly Supervised Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2851–2864, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2851 A Discrete Hard EM Approach for Weakly Supervised Question Answering Sewon Min1, Danqi Chen2,3, Hannaneh Hajishirzi1,4, Luke Zettlemoyer1,3 1University of Washington, Seattle, WA 2Princeton University, Princeton, NJ 3Facebook AI Research, Seattle, WA 4Allen Institute for Artificial Intelligence, Seattle, WA {sewon,hannaneh,lsz}@cs.washington.edu danqic@cs.princeton.edu Abstract Many question answering (QA) tasks only provide weak supervision for how the answer should be computed. For example, TRIVIAQA answers are entities that can be mentioned multiple times in supporting documents, while DROP answers can be computed by deriving many different equations from numbers in the reference text. In this paper, we show it is possible to convert such tasks into discrete la- tent variable learning problems with a precom- puted, task-specific set of possible solutions (e.g. different mentions or equations) that con- tains one correct option. We then develop a hard EM learning scheme that computes gradi- ents relative to the most likely solution at each update. Despite its simplicity, we show that this approach significantly outperforms previ- ous methods on six QA tasks, including abso- lute gains of 2–10%, and achieves the state- of-the-art on five of them. Using hard updates instead of maximizing marginal likelihood is key to these results as it encourages the model to find the one correct answer, which we show through detailed qualitative analysis.1 1 Introduction A natural setting in many question answering (QA) tasks is to provide weak supervision to deter- mine how the question should be answered given the evidence text. For example, as seen in Fig- ure 1, TRIVIAQA answers are entities that can be mentioned multiple times in supporting doc- uments, while DROP answers can be computed by deriving many different equations from num- bers in the reference text. Such weak supervi- sion is attractive because it is relatively easy to gather, allowing for large datasets, but complicates learning because there are many different spurious ways to derive the correct answer. It is natural to 1Our code is publicly available at https://github. com/shmsw25/qa-hard-em. Figure 1: Examples from two different question an- swering tasks. (Top) Multi-mention reading compre- hension. The answer text is mentioned five times in the given document, however, only the fourth span actually answers the question. (Bottom) Reading comprehen- sion with discrete reasoning. There are many poten- tial equations which execute the answer (‘4’), but only one of them is the correct equation (‘40-36’) and the others are false positives. model such ambiguities with a latent variable dur- ing learning, but most prior work on reading com- prehension has rather focused on the model archi- tecture and used heuristics to map the weak sig- nal to full supervision (e.g. by selecting the first answer span in TRIVIAQA (Joshi et al., 2017; Tay et al., 2018; Talmor and Berant, 2019)). Some models are trained with maximum marginal like- lihood (MML) …

Content truncated to 500 words