AAAI Publications, 2010 AAAI Fall Symposium Series

Font Size: 
Enhanced Visual Scene Understanding through Human-Robot Dialog
Matthew Johnson-Roberson, Jeannette Bohg, Danica Kragic, Gabriel Skantze, Joakim Gustafson, Rolf Carlson

Last modified: 2010-11-03


In this paper, we propose a novel human-robot-interaction framework for the purpose of rapid visual scene understanding. The task of the robot is to correctly enumerate how many separate objects there are in the scene and to describe them in terms of their attributes. Our approach builds on top of a state-of-the-art 3D segmentation method segmenting stereo reconstructed point clouds into object hypotheses and combines it with a natural dialog system. By putting a `human in the loop', the robot gains knowledge about ambiguous situations beyond its own resolution. Specifically, we are introducing an entropy-based system to spot the poorest object hypotheses and query the user for arbitration. Based on the information obtained from the human-to-robot dialog, the scene segmentation can be re-seeded and thereby improved. We present experimental results on real data that show an improved segmentation performance compared to segmentation without interaction.


Computer Vision; Segmentation; Human-Robot Dialog; Scene Understanding

Full Text: PDF