Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction

Book One

Volume

Proceedings of the AAAI Conference on Artificial Intelligence, 17

Natural Language Processing and Information Retrieval

We analyze a corpus of referring expressions collected from user interactions with a multimodal travel guide application. The analysis suggests that, in dramatic contrast to human-human interaction, the interpretation of referring expressions can be computed with very high accuracy using a model which pairs a highly impoverished notion of discourse state with a simple set of rules that are insensitive to the type of referring expression used. We attribute this unexpected result to the implicit manner in which the interface conveys the system’s beliefs about the operative discourse state, to which users appear to tailor their choice of referring expressions. This result offers new insight into the way that computer interfaces can shape a user’s language behavior, insights which can be exploited to bring otherwise difficult interpretation problems into the realm of tractability.