What Makes A Good Story? Designing Composite Rewards for Visual Storytelling

  • Junjie Hu Carnegie Mellon University
  • Yu Cheng Microsoft Dynamics 365 AI Research
  • Zhe Gan Microsoft Dynamics 365 AI Research
  • Jingjing Liu Microsoft Dynamics 365 AI Research
  • Jianfeng Gao Microsoft Research
  • Graham Neubig Carnegie Mellon University

Abstract

Previous storytelling approaches mostly focused on optimizing traditional metrics such as BLEU, ROUGE and CIDEr. In this paper, we re-examine this problem from a different angle, by looking deep into what defines a natural and topically-coherent story. To this end, we propose three assessment criteria: relevance, coherence and expressiveness, which we observe through empirical analysis could constitute a “high-quality” story to the human eye. We further propose a reinforcement learning framework, ReCo-RL, with reward functions designed to capture the essence of these quality criteria. Experiments on the Visual Storytelling Dataset (VIST) with both automatic and human evaluation demonstrate that our ReCo-RL model achieves better performance than state-of-the-art baselines on both traditional metrics and the proposed new criteria.

Published
2020-04-03
Section
AAAI Technical Track: Natural Language Processing