We present a novel approach to creating structured diagrams (such as flow charts or object diagrams) by combining an off-line sketch recognition system with the user interface of a traditional structured graphics editor. Our system, called UDSI (user-directed sketch interpretation), aims to provide drawing freedom by allowing the user to sketch entirely off-line using a pure pen-and-paper interface. The results of the drawing can then be presented to UDSI, which recognizes shapes, lines, and text areas that the user can then polish as desired. The system can infer multiple interpretations for a given sketch to aid during the user’s polishing stage. The UDSI program uses a novel recognition architecture that combines low-level recognizers with domain-specific heuristic filters and a greedy algorithm that eliminates incompatible interpretations.