AAAI

Association for the Advancement of Artificial Intelligence

Home > Proceedings / Proceedings of the AAAI Conference on Artificial Intelligence, 36 > No. 10: AAAI-22 Technical Tracks 10

Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

Authors

Steven Y. Feng

Language Technologies Institute, Carnegie Mellon University


Kevin Lu

University of Waterloo


Zhuofu Tao

University of California, Los Angeles


Malihe Alikhani

School of Computing and Information, University of Pittsburgh


Teruko Mitamura

Language Technologies Institute, Carnegie Mellon University


Eduard Hovy

Language Technologies Institute, Carnegie Mellon University


Varun Gangal

Language Technologies Institute, Carnegie Mellon University


Proceedings:

No. 10: AAAI-22 Technical Tracks 10

Volume

Issue:

Proceedings of the AAAI Conference on Artificial Intelligence, 36

Track:

AAAI Technical Track on Speech and Natural Language Processing

Downloads:

Download PDF

Abstract:

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

DOI:

10.1609/aaai.v36i10.21306


AAAI

Proceedings of the AAAI Conference on Artificial Intelligence, 36