Grounding Language in Spatial Routines

Stefanie Tellex, Deb Roy

This paper describes a spatial language understanding system based on a lexicon of words defined in terms of spatial routines. A spatial routine is a script composed from a set of primitive operations on sensor data, analogous to Ullman's visual routines. We hypothesize that a set of primitives that underlie spatial language could be used to succinctly express the meaning of spatial terms in a way that can be used to interpret natural language commands. This hypothesis is tested by using spatial routines to build a natural language interface to a real-time strategy game, in which a player controls an army of units in a battle. Spatial routines are used as a top down control mechanism for the system, providing a mapping from natural language commands to system behavior. Although the current implementation has significant limitations as revealed in an initial evaluation, we believe that the concept of spatial routines holds promise as a way to ground spatial language semantics in terms of embodied sensory-motor interaction.

Subjects: 13. Natural Language Processing; 18. Speech Understanding

Submitted: Jan 26, 2007

This page is copyrighted by AAAI. All rights reserved. Your use of this site constitutes acceptance of all of AAAI's terms and conditions and privacy policy.