An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-level Structural Information

No. 15: AAAI-21 Technical Tracks 15

Volume

Proceedings of the AAAI Conference on Artificial Intelligence, 35

AAAI Technical Track on Speech and Natural Language Processing II

In this paper, we focus on the problem of unsupervised image-sentence matching. Existing research explores to utilize document-level structural information to sample positive and negative instances for model training. Although the approach achieves positive results, it introduces a sampling bias and fails to distinguish instances with high semantic similarity. To alleviate the bias, we propose a new sampling strategy to select additional intra-document image-sentence pairs as positive or negative samples. Furthermore, to recognize the complex pattern in intra-document samples, we propose a Transformer based model to capture fine-grained features and implicitly construct a graph for each document, where concepts in a document are introduced to bridge the representation learning of images and sentences in the context of a document. Experimental results show the effectiveness of our approach to alleviate the bias and learn well-aligned multimodal representations.

10.1609/aaai.v35i15.17573