Learning and Transfering Roles in Multi-Agent MDPs

Contents

Many real-world domains contain multiple agents that play distinct roles in achieving an overall mission. For instance, in tactical battle scenarios, a tank fulfills a much different role than a foot soldier. When learning an overall multi-agent strategy, it is clearly advantageous to have knowledge of the agent roles and the typical behavioral characteristics associated with each role. The goal of this paper is to learn and transfer information about agent role structure in the setting of multi-task reinforcement learning. We present a hierarchical Bayesian framework for representing and learning a distribution over the unknown set of agent roles and policies from a sequence of related multi-agent RL problems. This knowledge is then transferred to new problems by treating the distribution as an informative prior over the roles and policies of individual agents. In particular, in this work, the prior is used to initialize policy-gradient search. We demonstrate the effectiveness of this role transfer method on a real-time strategy game in the context of a sequence of tactical battles, each involving different numbers and types of individual units. The results show that our approach is able to find a set of underlying roles and use them to significantly speed up learning on novel problems.