Be Inaccurate but Don’t Be Indecisive: How Error Distribution Can Affect User Experience

No. 1: AAAI-19, IAAI-19, EAAI-20

Volume

Proceedings of the AAAI Conference on Artificial Intelligence, 33

AAAI Technical Track: Humans and AI

System accuracy is a crucial factor influencing user experience in intelligent interactive systems. Although accuracy is known to be important, little is known about the role of the system’s error distribution in user experience. In this paper we study, in the context of background music selection for tabletop games, how the error distribution of an intelligent system affects the user’s perceived experience. In particular, we show that supervised learning algorithms that solely optimize for prediction accuracy can make the system “indecisive”. That is, it can make the system’s errors sparsely distributed throughout the game session. We hypothesize that sparsely distributed errors can harm the users’ perceived experience and it is preferable to use a model that is somewhat inaccurate but decisive, than a model that is accurate but often indecisive. In order to test our hypothesis we introduce an ensemble approach with a restrictive voting rule that instead of erring sparsely through time, it errs consistently for a period of time. A user study in which people watched videos of Dungeons and Dragons sessions supports our hypothesis.

10.1609/aaai.v33i01.33012604