This paper describes the spatial aggregation language and its applications. Spatial aggregation comprises a framework and a mechanism for organizing computations around image-like, analogue representations of physical processes in data interpretation and control tasks; it transforms a numerical input field to successively higher-level descriptions by applying a small, identical set of operators to each layer given a metric, neighborhood relation and equivalence relation. The spatial aggregation language provides two abstract data types (ADTs) -- neighborhood graph and field -- and a set of interface operators for constructing the transformations of the field. The language consists of a library of component implementations from which a user can mix-and-match and specialize for a particular application. The modular design of the ADTs supports language extensions and user control over tradeoffs such as efficiency vs. generality. We illustrate the use of the language with examples ranging from region growing in image analysis to trajectory grouping in dynamics interpretation. Programs for these different task domains can be written in a modular, concise fashion in the spatial aggregation language. The language allows users to isolate and express important computational ideas while hiding low-level details.