Automatic detecting neutral faces will improve the accuracy of face recognition/authentication, speaker ID analysis, and make it is possible to do facial expression analysis automatically. This paper describes an automatic system to find neutral faces in images by using location and shape features. Using these features, a window is placed in the detected and normalized face region. In that fashion, zones in this window correspond to facial regions that are loosely invariant from subject to subject. Within these zones, shape features in the form of histograms and ellipses are extracted. These features, in addition to more global distance measures, are input to a classifier to arrive at a neutral/non-neutral decision. The system has achieved an average detection rate of 97.2% based on 536 test images of 24 subjects. Tests on an independent image database of 64 images of 13 subjects achieved success rate of 95.3%, giving some indication of the robustness of the system.