Flat Multi-modal Interaction Transformer for Named Entity Recognition

Proceedings of the 29th International Conference on Computational Linguistics, pages 192–217 October 12–17, 2022. 192 Neuro-Symbolic Visual Dialog Adnen Abdessaied∗, Mihai Bâce, Andreas Bulling Institute for Visualization and Interactive Systems (VIS) University of Stuttgart, Germany adnen.abdessaied@vis.uni-stuttgart.de mihai.bace@vis.uni-stuttgart.de andreas.bulling@vis.uni-stuttgart.de Abstract We propose Neuro-Symbolic Visual Dialog (NSVD)1 —the first method to combine deep learning and symbolic program execu- tion for multi-round visually-grounded reason- ing. NSVD significantly outperforms existing purely-connectionist methods on two key chal- lenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demon- strate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accu- racy of 99.72% on CLEVR-Dialog —a relative improvement of more than 10% over the state of the art —while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against in- complete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes. 1 Introduction Modelled after human-human communication, vi- sual dialog involves reasoning about a visual scene through multiple question-answering rounds in nat- ural language (Das et al., 2019). Its multi-round nature gives rise to one of its unresolved key chal- lenges: co-reference resolution (Kottur et al., 2018; Das et al., 2019). That is, as dialogs unfold over time, questions tend to include more and more pronouns, such as “it”, “that”, and “those” that have to be resolved to the appropriate previously- mentioned entities in the scene. Co-reference reso- lution is profoundly challenging (Das et al., 2019; ∗*Corresponding Author. 1Project page: https://perceptualui.org/ publications/abdessaied22_coling/ Hu et al., 2017), even for models specifically de- signed for this task (Kottur et al., 2018). Existing models follow a purely connectionist approach and suffer from several limitations: first, they require large amounts of training data, which is prohibitive for most settings. Second, these models are not ex- plainable, making it difficult to troubleshoot their logic when co-references are incorrectly resolved. Finally, current models lack generalisability, in par- ticular for real-world dialogs that include incom- plete or inaccurate dialog histories, longer dialogs than those seen during training, or unseen question types. While neuro-symbolic hybrid models have proven effective as a more robust, explainable, and data-efficient alternative, e.g. for VQA (Yi et al., 2018), video QA (Yi et al., 2020), or commonsense reasoning (Arabshahi et al., 2021), they have not yet been explored for visual dialog. We fill this gap by proposing Neuro-Symbolic Visual Dialog (NSVD) —the first neuro-symbolic method geared towards visual dialog. Our method combines three novel contributions to disentangle vision and language understanding from reasoning: First, it introduces two different program genera- tors: a caption and a question program generator, the former of which induces a program …

Content truncated to 500 words