Acting on a Visual World: the Role of Perception in Multimodal HCI

Frederic Wolff, Antonella De Angeli, and Laurent Romary

Following the ecological approach to visual perception, this paper investigates multimodal referring acts in Human- Computer Interaction. Preliminary results from a simulation experiment allow to: (a) clarify the effect perceptual organization on multimodal communication; (b) provide guidelines for designing effective multimodal interfaces. Demonstrating that both the verbal and the gestural part of a referential act are influenced by perception, we confirm the need and the utility of taking into account perceptual organization to analyze referential expressions in a more robust way. As a conclusion, we show how these results can lead to an actual implementation of a gesture interpretation module directed by the reference analysis process.

This page is copyrighted by AAAI. All rights reserved. Your use of this site constitutes acceptance of all of AAAI's terms and conditions and privacy policy.