Hermina J. M. Tabachneck and H. A. Simon
People are faced with having to process a diversity of percepts, i.e., verbal, visual, etc. In learning and problem solving, these include text, diagrams, graphs, and mathematical representations. What types of mental representations and processes do people use to deal with these? We know much about verbal processes, but nonverbal mental processes have been more difficult to trace. Much of the research on visual representations has been on elementary processes; little has been done either on complex visual problem solving or on the interaction between verbal and visual representations in problem solving.. Non-verbal mental processes become more accessible to study if (1) one assumes computational reality of modality-specific representations; (2) one assumes that environment-initiated representations and memory-based representations use the same medium and processes; and (3) one has evidence that a subject working with a particular representation. If these three criteria are satisfied, we hypothesize, then concurrent verbal protocols can be used to trace the use of non-verbal as well as verbal representations in complex problem solving. A method is described for studying both visual and verbal processes and their interactions in the context of learning difficult concepts. The method entails presenting different formats of the same data to subjects in a between-subject design. We guarded against contamination from previous experience and gave all subjects the same domain information. To obtain information on subjects’ problem solving processes and computations, we recorded subjects’ interactions with the environment-initiated representations in detail. We also modeled the subjects’ behavior, but our computer models are not described in this paper. Using this methodology in the domain of economics we found many differences in behavior between subjects using different data formats. The findings strongly confirmed the hypothesis that subjects’ verbalizations reflect, and thereby reveal, their internal representations of the information.