CoNAN: A Complementary Neighboring-based Attention Network for Referring Expression Generation

Proceedings of the 28th International Conference on Computational Linguistics, pages 1952–1962 Barcelona, Spain (Online), December 8-13, 2020 1952 CoNAN: A Complementary Neighboring-based Attention Network for Referring Expression Generation Jungjun Kim∗ Korea University, Republic of Korea jj kim@korea.ac.kr Hanbin Ko Korea University, Republic of Korea hb ko@korea.ac.kr Jialin Wu University of Texas at Austin jialinwu@utexas.edu Abstract Daily scenes are complex in the real world due to occlusion, undesired lighting conditions, etc. Although humans handle those complicated environments well, they evoke challenges for ma- chine learning systems to identify and describe the target without ambiguity. Most previous research focuses on mining discriminating features within the same category for the target ob- ject. One the other hand, as the scene becomes more complicated, human frequently uses the neighbor objects as complementary information to describe the target one. Motivated by that, we propose a novel Complementary Neighboring-based Attention Network (CoNAN) that explicitly utilizes the visual differences between the target object and its highly-related neighbors. These highly-related neighbors are determined by an attentional ranking module, as complementary features, highlighting the discriminating aspects for the target object. The speaker module then takes the visual difference features as an additional input to generate the expression. Our qualita- tive and quantitative results on the dataset RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our generated expressions outperform other state-of-the-art models by a clear margin. 1 Introduction Generating referring expressions (Mao et al., 2016; Yu et al., 2016; Liu et al., 2017; Yu et al., 2017; Tanaka et al., 2019), which identify target objects with simple words and phrases in everyday discourse, has attracted attention from both computer vision (CV) and natural language processing (NLP) com- munities. With the rapid development of RNNs (Hochreiter and Schmidhuber, 1997; Cho et al., 2014; Bahdanau et al., 2014) and the emergence of transformers (Vaswani et al., 2017), machine learning sys- tems can generate linguistically correct expressions in most cases. However, the remaining issue in the referring expression generation (REG) field is to avoid ambiguities (i.e. the generated expression should refer to a unique target object). This issue becomes increasingly important when referring to an object in complex daily scenes where occlusion, undesired lighting conditions, complex formation of objects, occur regularly. This complex nature inhibits the system from mining the unique features automatically for the target object among its visually similar ones. Previous works have mainly investigated model architectures to generate less ambiguous expressions. Speaker-Listener models (Mao et al., 2016; Liu et al., 2017) are widely adopted to encourage a speaker to generate expressions that can be comprehended by listener model. Further, Yu et al. (2017) employs a reinforcer module to reward the system if the generated expression is bonded to the target object. In order to find discriminative features for the target object and generate less ambiguous expression, plenty of research utilizes the visual differences between the target objects and the objects that belong to the same category determined by an object detector. However, when multiple visually similar objects appear in the scene, mining …

Content truncated to 500 words