Currently, mast robot programming is done either by manual programming or by the "teach-by-showing "method using a teach pendant. Both of these methods have been found to have several drawbacks. We have been developing a novel method for programming a robot: the assembly-plan-from-observation (APO) method. The APO method aims to build a system that has threefold capabilities. It observes a human performing an assembly task, it understands the task based on this observation, and it generates a robot program to achieve the same task. This paper overviews our effort toward the realization of this method.