This paper describes the analysis and modeling of natural human behaviors on top of a routine based model of cognition. This theory hypothesizes that there exists in the brain a collection of built-in programs, which are coded with a fixed set of basic visual primitives and can be reprogrammed to carry out various visual tasks. Despite the general recognition of the value of such an approach and several pieces of biological evidence supporting it, no detailed successful model of visual routines has been built. Furthermore, no situated model has been described that acknowledges the specific advantages conferred on such an approach by the human body and elaborate eye-movement system. We built a real-time computer vision system that can simulate basic human visuo-motor behaviors, and provided a platform which facilitates the study of human world interaction as sequential assembly of perceptual primitives. Based on this platform, we break down natural behaviors by automatic subtask recognition, and model task planning with a Markov model. The Markov model ef- fectively allows subjects to anticipate by capturing the statistics of previous trials.