Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards

Authors

  • Yuhang Song University of Oxford
  • Jianyi Wang Beihang University
  • Thomas Lukasiewicz University of Oxford
  • Zhenghua Xu Hebei University of Technology
  • Shangtong Zhang University of Oxford
  • Andrzej Wojcicki Lighthouse
  • Mai Xu Beihang University

DOI:

https://doi.org/10.1609/aaai.v34i04.6040

Abstract

Intrinsic rewards were introduced to simulate how human intelligence works; they are usually evaluated by intrinsically-motivated play, i.e., playing games without extrinsic rewards but evaluated with extrinsic rewards. However, none of the existing intrinsic reward approaches can achieve human-level performance under this very challenging setting of intrinsically-motivated play. In this work, we propose a novel megalomania-driven intrinsic reward (called mega-reward), which, to our knowledge, is the first approach that achieves human-level performance in intrinsically-motivated play. Intuitively, mega-reward comes from the observation that infants' intelligence develops when they try to gain more control on entities in an environment; therefore, mega-reward aims to maximize the control capabilities of agents on given entities in a given environment. To formalize mega-reward, a relational transition model is proposed to bridge the gaps between direct and latent control. Experimental studies show that mega-reward (i) can greatly outperform all state-of-the-art intrinsic reward approaches, (ii) generally achieves the same level of performance as Ex-PPO and professional human-level scores, and (iii) has also a superior performance when it is incorporated with extrinsic rewards.

Downloads

Published

2020-04-03

How to Cite

Song, Y., Wang, J., Lukasiewicz, T., Xu, Z., Zhang, S., Wojcicki, A., & Xu, M. (2020). Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 5826-5833. https://doi.org/10.1609/aaai.v34i04.6040

Issue

Section

AAAI Technical Track: Machine Learning