Unified Vision-Language Pre-Training for Image Captioning and VQA

Authors

  • Luowei Zhou University of Michigan
  • Hamid Palangi Microsoft
  • Lei Zhang Microsoft
  • Houdong Hu Microsoft
  • Jason Corso University of Michigan
  • Jianfeng Gao Microsoft

DOI:

https://doi.org/10.1609/aaai.v34i07.7005

Abstract

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

Downloads

Published

2020-04-03

How to Cite

Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020). Unified Vision-Language Pre-Training for Image Captioning and VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 13041-13049. https://doi.org/10.1609/aaai.v34i07.7005

Issue

Section

AAAI Technical Track: Vision