Dutch Railways use a special train for the ultrasonic inspection of rails. The output of the ultrasonic scanning system installed on the train consists of echo images - so-called B-scans. The B-scans contain images of rail constructions, noise artifacts, and/or defects. Originally, all the images were interpreted and classified by an operator. Later, a simple rule-based classifier was build which could classify some of the images automatically. Recently, a new version of the train has been built capable of faster and more detailed rail inspection. This necessitated improvement of the automatic classification software. A prototype system has been developed which uses both a rule-based expert system and case-based reasoning (CBR) for the image classification. A hybrid architecture has been chosen because it satisfies the requirements better than systems based on one technique only. The paper describes the overall system design and presents the results of tests on real data. The future work necessary for the deployment of the system on the inspection train is outlined.